Log “needs help” detector to detect STUCK and usability issues

JackMostow commented 3 years ago

Shashank Priyadarshi [ ] Log “needs help” detector and stack trace from judithodili/RoboTutor_Help_Intervention

JackMostow commented 3 years ago

Searching for "trigger" finds some events that sound worth logging:

JackMostow commented 3 years ago

RoboTutor may already be closer to logging STUCK (and similar events) than I knew, thanks to its triggers for detecting them in order to perform interventions! The interventions depended on getting help from another student, so we don't want to execute them interventions (at least for now), just log them. They used some information derived from RoboTutor log data, e.g. the 95th percentile time it took kids to do a given action (activity? item? attempt?). I found constants to replace them starting at line 558 in /util/src/main/java/cmu/xprize/util/TCONST.java.

Does log.wtf go to RoboTutor's verbose log file, or just to the Android Studio runtime log?

Do we need to enable config.json to set CONFIG_INTERVENTION to false, not just hide the help button? Or would setting CONFIG_INTERVENTION to false disable the triggers?

util/src/main/java/cmu/xprize/util/consts/INTERVENTION_CONST.java

11 | public static final boolean CONFIG_INTERVENTION = true;
-- | --

public void triggerIntervention(String type) {
    GlobalStaticsEngine.setCurrentTutorType("SPELL");
    **Log.wtf("trigger", "triggering intervention: " + type);**
    Intent msg = new Intent(type);
    _bManager.sendBroadcast(msg)

public void onReceive(Context context, Intent intent) {
    // for turning off intervention mode
    if (**!CONFIG_INTERVENTION**) {
        return;
    }

    **Log.wtf("trigger", "REceived trigger: " + intent.getAction());**

JackMostow commented 3 years ago

To help us interpret logged trigger firings, this is when each type of trigger fires:

"gesture": "the student tries the wrong gesture",
"stuck": "the student tries, can't try a question e.g. doesn't tap bubble in bpop",
"hesitate": "the student does nothing at all. should only trigger on first problem"
"failure": "the student fails on the first try for X problems" -- from https://github.com/RoboTutorLLC/RoboTutor_2020/app/src/main/assets/tutors/intervention_descriptor.json

iamsh4shank commented 3 years ago

Hey @JackMostow can you tell me where we are getting this issue

JackMostow commented 3 years ago

@robustTechie I'm not sure what your question means, so here are two answers:

The motivation for this issue is to find and fix STUCK bugs where an activity can't advance and the user must tap Back to escape. Unlike CRASH bugs, which are automatically detected and logged at runtime along with a stack trace, obvious to kids, and easy to recover from by relaunching RoboTutor, STUCK bugs are not immediately obvious to kids, they recover eventually by tapping the Back button, and we detect them only after the fact by analyzing logs to find cases where kids tap Back after a long hiatus. Note that this sense of STUCK is RoboTutor-centric in that it refers to a state of the app, namely inability to advance in the activity.

In contrast, the "stuck" and "hesitate" triggers in the code added to support Judith Uchidiuno's thesis experiment but not currently used in RoboTutor are child-centric in that they refer to the kid's behavior. They overlap with the RoboTutor-centric STUCK bug in the sense that they may cause the same phenomenon of a long hiatus followed by tapping the Back button. I normally avoid using the same term to mean two different things, but unfortunately we were already using both STUCK and "stuck," so I'm using STUCK to refer to the program bug and "stuck" to refer to the kid's state. If your question was asking where are we getting the triggers, they come from the code added for Judith's thesis experiment, which uses timers to detect and broadcast events like TCONST.I_TRIGGER_STUCK using activity-type-specific thresholds documented in intervention_descriptor.json, e.g.: "bpop": { "gesture": "anything but tap", "stuck": "10s", "hesitate": "10s", ...

Judith's goal was to identify cases where the kid needs help, and address them by referring the kid to a peer who can likely help because he or she has coped with it before. In contrast, the goal of this issue is to find and fix bugs where RoboTutor cannot advance in an activity. Our goal is to detect such bugs in realtime and log them with a stack trace to help diagnose and fix them.

Does this long reply answer your question?

iamsh4shank commented 3 years ago

Yeah thanks @JackMostow it answered my question, just one more doubt -

Our goal is to detect such bugs in realtime and log them with a stack trace to help diagnose and fix them.

So I need to put a LOG for this type of issue right?

JackMostow commented 3 years ago

Yes, both as an event in the verbose log, and as a file analogous to a CRASH log with an error message and a stack trace, except find some logical analogue for the error message -- whatever seems like it might help us understand the context and isn't already included in the stack trace. Perhaps the tutor name or type? To make it easy to aggregate multiple occurrences of the same event, the file should not include timestamps or anything else specific to an individual instance.

VishnuSanal commented 3 years ago

Hey @JackMostow!

Just to clarify, I should log (in JSON & in LogCat) whenever a stuck, hesitation or gesture happens, from these respective triggers:

TimerMaster#triggerGestureTimer TimerMaster#triggerStuckTimer TimerMaster#triggerHesitationTimer

Am I right?

Thanks!

JackMostow commented 3 years ago

I think so.

What and where is LogCat?
Can you log in JSON using the same logging function used elsewhere so as to get the same format, e.g. including timestamp?
Does “gesture” trigger only for gestures not interpreted by the activity, e.g. tapping on a non-target?
Can you include trigger details when logging these events, such as the type of gesture and the activity and state in which the event occurred? Please send me a concrete example of how you propose to log each event, to make sure we’re on the same page. Thanks! – Jack From: Vishnu Sanal. T @.> Sent: Wednesday, May 12, 2021 4:09 PM To: RoboTutorLLC/RoboTutor_2020 @.> Cc: Jack Mostow @.>; Mention @.> Subject: Re: [RoboTutorLLC/RoboTutor_2020] Log “needs help” detector to detect STUCK and usability issues (#32)

Hey @JackMostowhttps://github.com/JackMostow!

Just to clarify, I should log (in JSON & in LogCat) whenever a stuck, hesitation or gesture happens, from these respective triggers:

TimerMaster#triggerGestureTimerhttps://github.com/RoboTutorLLC/RoboTutor_2020/blob/cc9d095235dfe8d6fbbd3b425a50bebbd0e4aa01/util/src/main/java/cmu/xprize/util/TimerMaster.java#L131 TimerMaster#triggerStuckTimerhttps://github.com/RoboTutorLLC/RoboTutor_2020/blob/cc9d095235dfe8d6fbbd3b425a50bebbd0e4aa01/util/src/main/java/cmu/xprize/util/TimerMaster.java#L80 TimerMaster#triggerHesitationTimerhttps://github.com/RoboTutorLLC/RoboTutor_2020/blob/cc9d095235dfe8d6fbbd3b425a50bebbd0e4aa01/util/src/main/java/cmu/xprize/util/TimerMaster.java#L105

Am I right?

Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/RoboTutorLLC/RoboTutor_2020/issues/32#issuecomment-840063466, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC64MWKFCOKWL3BCGIZO4Q3TNLN35ANCNFSM44GAZZXQ.

JackMostow commented 3 years ago

The activity-specific thresholds I mentioned are in RoboTutor_2020/util/src/main/java/cmu/xprize/util/TCONST.java l. 559ff: public static final int FAILURE_COUNT_BPOP = 9; public static final Long HESITATE_TIME_BPOP = 6000L; // NOTE THAT THIS IS NOT RESETTING public static final Long STUCK_TIME_BPOP = 20000L; public static final Long GESTURE_TIME_BPOP = 6000L;

public static final int  FAILURE_COUNT_AKIRA = 9;
public static final Long HESITATE_TIME_AKIRA = 10000L;
public static final Long STUCK_TIME_AKIRA = -1L; // not applicable...
public static final Long GESTURE_TIME_AKIRA = 10000L;

public static final Long HESITATE_TIME_SPELL = 15000L;
public static final Long STUCK_TIME_SPELL = 18000L;
public static final Long GESTURE_TIME_SPELL = 13000L;

public static final Long HESITATE_TIME_PICMATCH = 18000L;
public static final Long STUCK_TIME_PICMATCH = 20000L;
public static final Long GESTURE_TIME_PICMATCH = 13000L;

public static final Long HESITATE_TIME_WRITE = 18000L;
public static final Long STUCK_TIME_WRITE = 20000L;
public static final long GESTURE_TIME_WRITE = 18000L;

public static final Long HESITATE_TIME_NUMCOMPARE = 9000L;
public static final Long STUCK_TIME_NUMCOMPARE = 12000L;
public static final Long GESTURE_TIME_NUMCOMPARE = 9000L;

JackMostow commented 3 years ago

Fixed in squashed merge of pull request #42

RoboTutorLLC / RoboTutor_2020

Log “needs help” detector to detect STUCK and usability issues #32