RoboTutorLLC / RoboTutor_2020

Other
3 stars 1 forks source link

Allow runtime switching of MATRIX_FILE (changed 2/1/2022 from Let config.json set MATRIX_FILE) #54

Closed JackMostow closed 1 year ago

JackMostow commented 3 years ago

To replicate logged bugs, we need to set MATRIX_FILE to dev_data.json, last modified 9/21/2018 and used at XPRIZE sites, instead of dev_data.open.json, last modified 4/14/2019 and used in later versions of RoboTutor including the current one.

Please let config.json override the value of MATRIX_FILE set in robotutor.java.

Alternatives: Doing it in debug.json might be cleaner, but we need the same activity behavior with debug.json as without it. Specifying a switch for which MATRIX_FILE to use would prevent specifying a non-existent file, but take more work and be less clear. The value of MATRIX_FILE has had several different values, so we might want to change it again. For instance, we might want a stable version to use for demos, and a less stable version for testing and debugging.

JackMostow commented 3 years ago

Will facilitate #31 and #29.

JackMostow commented 3 years ago

set_MATRIX_FILE could be the branch name for this issue.

JackMostow commented 3 years ago
  1. @eviishondell is double-checking our belief that some stories have different tutor IDs now than in our table of logged cd2 bugs.
  2. Can MATRIX_FILE really be changed from a compile-time constant to a runtime variable loaded from config.json? Not if the code compiles the activity matrix into an asset file loaded at compile time or first launch rather than when it reads config.json. Does it?
JackMostow commented 3 years ago

Looks like compile-time:

Matrix File Location: tutors/activity_selector/assets/data/[lang (en or sw)]/dev_data.open.json, e.g. https://github.com/RoboTutorLLC/RoboTutor_2020/blob/development/app/src/main/assets/tutors/activity_selector/assets/data/sw/dev_data.open.json -- i.e. in GitHub, not (apparently) on Android device

If so, let's hold off because the task isn't as simple as I thought. And if we're lucky, we don't need it anyway (i.e. if tutor IDs didn't change as we feared).

JackMostow commented 3 years ago

@eviishondell @1535373 Do you have any "smoking tutorID" instances of tutors whose ID or data source definitely differs from cd2 (2.7.7.1) to the newer version(s) you've been testing, e.g. 3.3.2.1 (or 3.3.1.2, I forget which)? This question is important because if the answer is yes, a bug recipe based on data from 2.7.7.1 may be testing the wrong tutor in newer versions.

Keep in mind that even if it's testing the right tutor, the logged bug may have already been fixed in newer versions, but at least the recipe ought to bring up the same item that used to trigger the bug.

I'm hoping the answer is no, i.e. recipes based on logged bugs still work in newer versions of RoboTutor. If they don't, we still have a fallback: make a special test apk with the new code but the old matrix file. It would be less convenient, because we'd have to keep generating new ones if we want to keep testing the latest code. But it should still do the trick.

JackMostow commented 2 years ago

2/1/2022: Implementing Multi-Armed Bandit will involve switching among alternative values of MATRIX_FILE.

JackMostow commented 1 year ago

MAB will select arm and matrix file at launch.