BUG: Help! RoboTutor HANG while installing assets. Does it for you?

[JackMostow]

Madelein and Rohan - Help! RoboTutor RoboTutor-2022-06-13-stale-cached-assets-2022-06-29-v3.5.0.1.apk first launch hangs while installing English_StoriesAudio_1.1.0.zip: Screenshot_20220826-161046.png So moved English_StoriesAudio_1.1.0.zip out of 2022 Download RoboTutor and relaunched RoboTutor. Hung immediately. So moved all the files out and relaunched. Same problem. Also with RoboTutor-screenrecording-rohit-2022-06-27-v3.5.0.1.apk from previous commit from development branch. I'm getting this problem on the same Android 6 tablet I've successfully installed many previous versions of RoboTutor on. It's the only working Android tablet I brought with me to Chicago. (I also brought an Android 8.1 tablet but Play Store doesn't work on it so I can't update its outdated Google Drive or install DriveSync to download RoboTutor.) What's the newest development branch commit whose apk you can install and launch including installing assets on first launch?

[madeleinvillegas]

Dr. Mostow,

I've also had the same problem a few weeks or months back. I think the apk from my last commit (RoboTutor-2022-03-26-Madelein-Crash-Log-Filenames-Final-2022-03-25-v3.5.0.1.apk) seems to be the newest apk that works.

[JackMostow]

Madelein - Very useful to know! I’ll try RoboTutor-2022-03-26-Madelein-Crash-Log-Filenames-Final-2022-03-25-v3.5.0.1.apk.

Can you try to replicate and track down the bug by running in Android Studio? Here are a few ideas for how, but you may have better ones! They’re just in the order I thought of them, but you may want to reorder them by the time to try them and your estimate of their odds of success.

1. If the bug started after your last commit, check if any of the code changes in the next commit could have caused it.

2. I tried removing asset files from the Download folder to speed up asset installation and localize the problem, but it still occurred.

3. It might be necessary to eliminate the unzipped assets first in case one or more of them is causing the problem. Some of them go into the robotutor_assets folder, but others might be hidden somewhere I didn’t find.

4. It may help to log asset loading in the verbose log to locate the point where RoboTutor hangs, in particular whether it occurs during or after installing assets.

5. It may also help to compile and test a DEBUG version. It will be slower but hopefully more informative.

6. Running in Android Studio and inspecting its detailed log output (not to be confused with the verbose log file output by RoboTutor) will likely locate the bug faster than running the app as a black box.

Thanks!! - Jack

Sent from my iPhone so beware typos

On Aug 26, 2022, at 9:28 PM, Madelein Villega @.***> wrote:

 Dr. Mostow,

I've also had the same problem a few weeks or months back. I think the apk from my last commit (RoboTutord-2022-03-26-Madelein-Crash-Log-Filenames-Final-2022-03-25-v3.5.0.1.apk) seems to be the newest apk that works.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[JackMostow]

@madeleinvillegas - Yes, RoboTutor-2022-03-26-Madelein-Crash-Log-Filenames-Final-2022-03-25-v3.5.0.1.apk committed by Travis CI on Mar 25, 2022 successfully launched and installed assets. development : (https://github.com/RoboTutorLLC/RoboTutor_2020/commit/3c5cd5587b744177f6d8238971760978f679cdb0) : (2022-03-26.02:06:53)

Was it compiled from the same source code committed to the development branch on Mar 31, 2022? Added Session ID and Sequence number to the filename of crash files (https://github.com/RoboTutorLLC/RoboTutor_2020/pull/89

[JackMostow]

The next commit to development was by Rohit on 6/7/2022: Added Screen Recording (https://github.com/RoboTutorLLC/RoboTutor_2020/pull/95)

It had 168 changed files with 2,381 additions and 2,203 deletions, which will make it hard to tell which if any caused the bug:

• migrated project to AndroidX, added HBRecorder as dependency

• Single recording of session done

• Screen Recording settings updated for smaller video recording file size

• screen recording config based on config.json hooks

• Screen Recording Video directory and name changes

• update config.json hooks

• on activities that use mic, disable activity audio recording and pause session audio recording

There was one intervening commit to the apk branch: development : (https://github.com/RoboTutorLLC/RoboTutor_2020/commit/ba6cdec8f10076fdaedd40bce62034369ac7165a) : (2022-04-13.17:41:54) apk Travis CI committed on Apr 13 1 parent ba6cdec commit 037660fb0c6c5f76326396545da201b9a0d78114 Showing 1 changed file with 0 additions and 0 deletions. BIN +65.2 MB RoboTutor-2022-03-29-Inserting-NSP-Questions-2022-04-13-v3.5.0.1.apk

We can see if it has the same bug, but it might not tell us much if its source code wasn't committed to the development branch.

[JackMostow]

Good news! Till now I couldn't use Google Drive, Drive Sync, or Play Store on the one 8.1 Google Pixel C tablet I took to Chicago because it had an outdated copy of Google Play Services whose Update function hung while "pending". So I'm happy to report that I found a recent (8/20/2022) version (22.32.12 040400-466264887) of Google Play Services at https://google-play-services.en.uptodown.com/android/download (not a Google site!), which I was able to download, install, and open, which got Play Store to work so I could install Google Drive, DriveSync, Cx File Explorer, and the same RoboTutor apk and assets as on the 5 Google Pixel C tablets I sent to Bagamoyo. Now I'll be able to test whatever configuration we want to use there.

I no longer believe that ideas 1-3 in my 8/27/2022 comment are useful. The assets are presumably OK, because they load fine in RoboTutor-2022-03-29-Inserting-NSP-Questions-2022-04-13-v3.5.0.1.apk. But ideas 4, 5, and 6 still seem useful in tracking down and fixing the HANG bug in order to use screen recording and other new features.

[JackMostow]

@madeleinvillegas - Find way to detect "soft hang" -- see Hang doctor: runtime detection and diagnosis of soft hangs for smartphone apps. When triggered, dump stack trace. For debugging in Android studio, just put breakpoint at trigger and inspect stack and other relevant state.

[JackMostow]

To debug in Android Studio, compile DEBUG version and run tethered. Low-tech version of trigger: launch RoboTutor in Android Studio. When your patience runs out, interrupt it to see where it is. To test your patience less, hide the .zip assets in another folder. Beware that already unzipped assets may hide where you can't see them.

[JackMostow]

@madeleinvillegas - Here are some resources that might be useful.

Diagnosing ANRs:

... The ANRs could be solved easily as the Android Interface Provide Efficient Tools to remove them! Finding them may include (but are not limited to):

The app is doing slow operations which involves the I/O operation of the operating system. The main thread is doing a synchronous binder call to a different process, which another process is taking an extended time to return. The main thread is during a deadlock with another thread, either in your process or via a binder call. the most thread isn’t just expecting an extended operation to end but is during a deadlock situation. The app is doing an extended calculation on the basic thread. Using the Android Strict Mode (API Level > 9)

Starting API Level 9 using StrictMode will help you discover certain accidental I/O operations on the most thread while you’re developing your app. you’ll use StrictMode at the appliance or activity level. Read more about how to use StrictMode here.

Moreover, you can also find about the ANRs by pulling a Traces File from Android ADB as Follows:

adb root adb shell ls /data/anr adb pull /data/anr/

... View More Details about ANR by visiting Here.

[JackMostow]

@madeleinvillegas -

1. How are you feeling? Better, I hope!
2. What progress on the HANG bug since we met yesterday (8/29)?
3. (When) are you available today to resume debugging together? Thanks! - Jack

[JackMostow]

I got the same crash we had when I ran the stale cached assets apk. When I ran the screen recording apk, the logcat of Android Studio just kept going even when RoboTutor has already hanged. So I couldn’t find where the problem is.

RoboTutor-2022-06-13-stale-cached-assets-2022-06-29-v3.5.0.1.apk crashes

RoboTutor-screenrecording-rohit-2022-06-27-v3.5.0.1.apk hangs for Madelein (in middle of loading English assets) but the logcat of Android Studio keeps going

RoboTutor-screenrecording-rohit-2022-06-27-v3.5.0.1.apk doesn't hang for Jack if run without any assets in the Download folder.

[ ] So Jack download more assets -- especially the English stories audio -- to see if it hangs.

[JackMostow]

Madelein [ ] try in Android Studio:

1. Run RoboTutor-screenrecording-rohit-2022-06-27-v3.5.0.1.apk with no assets. Does it hang? No, but it crashes.
2. Put breakpoints or trace asset unzipping to localize when hang starts. Beware that it may be in a different thread.
3. Find out (in a CONST file) where assets get unzipped to. Is it only robotutor_assets, or somewhere else as well?
4. Run 6-29 apk with no assets. Does it hang?

[madeleinvillegas]

2. if (tutorId.contains(".read") || tutorId.contains(".echo") || tutorId.contains(".parrot") || tutorId.contains(".reveal")) is where RoboTutor crashes. tutorId is a null which breaks. The line is part of the method createAndLaunchTutor(String tutorName, String features, String tutorId, defdata_tutor dataSource, String matrix). When this method was called, tutorId is assigned to have a NULL value which likely causes the CRASH. But, when I checked the 2022-03-31 commit (working apk), the same line appears
3. I believe it is only robotutor_assets. There seems to be a sdcard/Download/RoboTutor/assets/story_questions for story audio but there is no assets/story_questions folder inside RoboTutor.
4. Run 6-29 apk with no assets. Does it hang? No, it crashes with the same error

[JackMostow]

@madeleinvillegas - The plot thickens!

2. That "if" statement is how screen recording disables audio recording when screen-recording activities that use the microphone. The culprit is whatever calls createAndLaunchTutor with a null tutorId. Where does the null value originate? For instance, is RoboTutor trying to look up a tutorname missing from the activity matrix? If we understand why, we should be able to fix it properly, not just insert a null value check as a bandaid to prevent the crash.
3. There's no RoboTutor/assets folder on my tablet, I wonder where it comes from.
4. Interesting ;-). Thanks! - Jack

[JackMostow]

Madelein says "Looking at the changes from 2022-03-31 to 2022-06-07, tutorId was assigned a NULL value in the launch method. But the debugger doesn’t show RoboTutor going through the said launch method. (Unless it is in a different thread that was skipped by the debugger)"

[madeleinvillegas]

If the RoboTutor-screenrecording-rohit-2022-06-27-v3.5.0.1.apk and RoboTutor-2022-06-13-stale-cached-assets-2022-06-29-v3.5.0.1.apk apks both have the changes from Screenrecording rohit, these two apks not working make sense.

When I ran 2022-06-07 commit Added Screen Recording (with and without assets) in Android Studio, I did not get a crash nor hang when installing.

When I ran 2022-06-28 commit Screenrecording rohit (without assets) in Android Studio, I got a HANG at glyphs.zip. The logcat kept on going even after the hang, but I think the hang still occurred at this line if (tutorId.contains(".read") || tutorId.contains(".echo") || tutorId.contains(".parrot") || tutorId.contains(".reveal")

Looking at the debugger, tutorId being assigned NULL at the was never a problem since tutorId was never used in a substantial way. The launch method has the value of tutorId (it is never called or it may be called but after the crash) but not the createAndLaunchTutor method in which the program runs through.

I think it is safe to say that the bug is in Screenrecording rohit commit.

[madeleinvillegas]

Stack trace:

2022-09-08 22:38:13.829 10049-10049/cmu.xprize.robotutor E/AndroidRuntime: FATAL EXCEPTION: main
    Process: cmu.xprize.robotutor, PID: 10049
    java.lang.NullPointerException: Attempt to invoke virtual method 'boolean java.lang.String.contains(java.lang.CharSequence)' on a null object reference
        at cmu.xprize.robotutor.tutorengine.CTutorEngine.createAndLaunchTutor(CTutorEngine.java:414)
        at cmu.xprize.robotutor.tutorengine.CTutorEngine.startSessionManager(CTutorEngine.java:261)
        at cmu.xprize.robotutor.RoboTutor.onStartTutor(RoboTutor.java:824)
        at cmu.xprize.robotutor.RoboTutor.startEngine(RoboTutor.java:796)
        at cmu.xprize.robotutor.RoboTutor.onServiceReady(RoboTutor.java:744)
        at cmu.xprize.util.TTSsynthesizer.onInit(TTSsynthesizer.java:68)
        at android.speech.tts.TextToSpeech.lambda$dispatchOnInit$0$TextToSpeech(TextToSpeech.java:890)
        at android.speech.tts.TextToSpeech$$ExternalSyntheticLambda17.run(Unknown Source:4)
        at android.speech.tts.TextToSpeech.dispatchOnInit(TextToSpeech.java:899)
        at android.speech.tts.TextToSpeech.access$900(TextToSpeech.java:79)
        at android.speech.tts.TextToSpeech$Connection$SetupConnectionAsyncTask.onPostExecute(TextToSpeech.java:2280)
        at android.speech.tts.TextToSpeech$Connection$SetupConnectionAsyncTask.onPostExecute(TextToSpeech.java:2240)
        at android.os.AsyncTask.finish(AsyncTask.java:771)
        at android.os.AsyncTask.access$900(AsyncTask.java:199)
        at android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:788)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loopOnce(Looper.java:226)
        at android.os.Looper.loop(Looper.java:313)
        at android.app.ActivityThread.main(ActivityThread.java:8663)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)

[JackMostow]

Initial launch should load assets before choosing an activity and setting tutorID. So why is it testing tutorId while still loading?! Does loading assets occur in a different thread or (now deprecated) AsyncTask? Is picking and starting a tutor the main thread, and loading assets another thread aka aynchronous task? Or vice versa? Shouldn't RoboTutor finish loading assets before picking and starting any activities, at least in a content area whose assets it hasn't finished loading? It could be faster to continue loading assets in the background while starting an activity that didn't need them, e.g. in a different content area. But assets aren't (all) specific to a particular content area. So one fix may be to find where the asset loader is called and do it in foreground instead of background.

Instead (or besides), what if we just put a null check into createAndLaunchTutor? If so, what would happen next instead of current HANG?

Also, what to do about null tutorId?

WHAT IF: asset loading is actually finishing, but tutorId is null for some other reason?

BTW, why (and when) is tutorId null? Presumably because it never got set, because whatever sets it didn't get called. So where is it set? cd2 activity selector picks from 3 options (current, next, redo). (cd1 lets kid pick content area, then picks activity at kid's position in selected content area.) What if one of the 3 options happens to be null? But HANG precedes picking, so that's not the problem. Why is RoboTutor trying to launch a tutor without picking it??

[JackMostow]

deftutor is default tutor, apparently null in our case. StartSessionManager tries to create and launch it, but it's null, which fails. Is asset loading a red herring? HANG occurs even with no assets, implying that it occurs after loading assets. It makes sense that a newly launched tutor might need to have a default tutor to run. How can we ensure that the default tutor is not null? Or what should we do if it is? Should we bandaid the bug by specifying a somewhat arbitrary tutor to use instead if tutorId is null? If not, where should StartSession go if it doesn't know which tutor to launch? Perhaps we still need to ensure that asset loading finishes, but it might not be the actual culprit! Stay tuned....

[JackMostow]

Curious experiment: if tutorId is null, just set it to some arbitrary activity to see what happens. For a brand new user with no student model (i.e. position in matrix), perhaps the default tutorId is null, and any tutorId is better than none?

[madeleinvillegas]

Can confirm that the assets finishes installing/loading without crashing. It just looks like a hang because there are no GUI indicators from the time the assets finish loading until the main menu of RoboTutor. No activities get picked until after loading assets.

This is how the program flows: startSessionManager calls createAndLaunchTutor where tutorId is NULL A tutor is created with the tutorName a couple of things happen until launch gets called where a tutorId is created, it then goes to a switch-case which calls createAndLaunchTutor where tutorId is not null anymore

Adding a null check seemed to work, I was able to load with (just two zip files) and without assets. I do not think having an arbitrary tutorId will work since the first time the createAndLaunchTutor is called, it creates a tutor based on the tutorName, we might confuse the program by providing a tutorName and tutorId because they both go through the same constructor.