search

RoboTutorLLC / RoboTutor_2019

Main code for RoboTutor. Uploaded 11/20/2018 to XPRIZE from RoboTutorLLC/RoboTutor.
Other
7 stars 4 forks source link

1.8.9.1 ECHOWORDS mode to encourage fluent reading #348

Open JackMostow opened 6 years ago

JackMostow commented 6 years ago

This issue was originally part of Revise READ UI/UX tracking #157:

Problem: get READ’s “blame the kid” strict left-to-right policy not to encourage kids to, read, one, word, at, a, time, unlike Project LISTEN’s Reading Tutor, which used “chase the kid” to track the kid through the text.

Solution:

  1. Rename RoboTutor's current ECHO as REREAD because it rereads the sentence after the kid reads it.
  2. Add an ECHO mode that works more like its namesake, Project LISTEN's Reading Tutor's ECHO mode.
  3. Relax the strict left-to-right policy to tolerate occasional rejected or skipped words.

Currently RoboTutor underlines the word it expects next. It turns words green if accepted and red if not. Requiring acceptance of each word before advancing to the next word discourages fluent reading due to false rejections.

Change this behavior as follows:

  1. Modify position tracking to tolerate an individual missed word but not 2 in a row. a. MultiMatch aligns ASR output to text to minimize total penalties for mismatches and jumps. It may or may not be the right mechanism to tolerate isolated missed words, e.g. by aligning only to i, i+1, or i+2. b. The policy means that the expected next word can skip from word i to i+2 but not further. c. Underline the expected next word to indicate the current position i. d. The current position should not move backwards from word i to j < i because it would be confusing. e. The current position can skip from i to i+2 if the ASR output ends with ... word i-1 word i+1.

  2. Color words green when accepted, red when rejected, and (still) black when skipped. a. Or should skipped words turn red? b. It's ok to visibly accept and credit a previously rejected word, but not to rescind credit once granted.

  3. When the reader pauses (0.5 sec) before reading word i, echo word(s) j..i-1, where word j-1 was the last echoed word, and turn each word green after echoing it. When RoboTutor reads just one word, use its isolated narration for clarity. Otherwise use the appropriate subsequence of its human narration. Note that: a. Reading, word, by, word will echo a word 0.5 sec after it's spoken, but wait patiently for the next word. b. Reading a phrase at a time will echo the phrase 0.5 sec after it's spoken.

  4. Treat a tap almost like reading the current word: a. A tap after a hesitation will just read (and credit) the current word i. b. A tap with no hesitation before word i will reread the word(s) j..i. I don't expect this case to occur often.

Notes:

  1. Project LISTEN's Reading Tutor changed the background color of each word to yellow while reading it, to show its position. This would be a nice feature if there's time, to make clear which word is being read.

  2. Project LISTEN's Reading Tutor deferred crediting words green rather than turn them green right away. We can implement that behavior if the behavior above causes kids to wait for each word to turn green before reading the next word.

    1. Replacing underlining with RoboFinger would be cute but is not worth the effort, especially if it requires representing the position of each text word on the screen.
JackMostow commented 6 years ago

Actually we can dispense with REREAD and just use ECHO, provided we modify ECHO to reread the entire sentence if the kid read it disfluently, in which case rereading it scaffolds comprehension.

JackMostow commented 6 years ago

@octavpo - can you implement ECHO by August 28 when I return?

JackMostow commented 6 years ago

@octavpo - Let me rephrase that request more emphatically:

XPRIZE weights reading 60%, ECHO tries to addess severe problems in RoboTutor's current ASR-driven assisted oral reading, and is important to kid-test ASAP.

So please implement ECHO in time for me to try out on August 29. If you have any questions before then, please email them to me.

I'll be offline Friday and Saturday PST, but will reply ASAP. I'll be travelling all day Tuesday, August 28.

Thanks. - Jack

JackMostow commented 6 years ago

Octav email of Saturday, September 01, 2018 6:29 PM:

... I'll try to put more details on the Github issue, but in short I found that in my tests most cases when a word is marked red are due to the fact that works are recognized and lost rather than because they're not recognized correctly. So I'm trying to fix that first.

Jack email of Sat 9/1/2018 9:47 PM:

“Words are recognized and lost” is very interesting. Based just on your own testing, or can you tell from the verbose logs from use in the field? How?

Please send a real example. From your brief description, I suspect a bug in the MultiMatch process that aligns ASR output against the text to read – not in MultiMatch itself, but perhaps something like omitting utterances from the speech aligned against the text if newer utterances have come in.

JackMostow commented 6 years ago

@octavpo - #357 said: READ often fails to respond to oral reading miscues. Why? a. The speech is not loud enough to pass the detection threshold, perhaps due to a noisy environment. b. The ASR recognizes something but RoboTutor doesn't respond.

Note that a word turns red only if RoboTutor detected speech, i.e. not in case a.

JackMostow commented 6 years ago

@judithodili suggested that lowering the speech detection threshold would make ASR hallucinate more. @judithodili and @amyogan - What if RoboTutor detects no speech for [3] seconds?

Options:

a. Don't supply the word automatically, but cue the kid to tap, e.g. have RoboFinger tap the audio icon +: keeps RoboTutor from advancing without the kid doing anything +: clearer than just displaying the audio icon +: simple

b. Treat the word as misread, i.e. turn it red. +: if kid attempted the word, responds visibly if not immediately -: if kid was silent, might cause confusion or frustration due to too much time pressure

c. If RoboTutor still detects no speech, invoke the 2-strike rule, i.e. supply the word. +: alleviate frustration caused by RoboTutor failing to detect kid's attempts to read the word -: lets RoboTutor keep reading even if the kid says nothing

I favor option a. Do you concur?

amyogan commented 6 years ago

Yes, I like keeping the kids actively involved!

JackMostow commented 6 years ago

@octavpo - Please implement option a.

octavpo commented 6 years ago

Please send a real example. From your brief description, I suspect a bug in the MultiMatch process that aligns ASR output against the text to read – not in MultiMatch itself, but perhaps something like omitting utterances from the speech aligned against the text if newer utterances have come in.

The problem is not the MultiMatch, which is pretty much disabled, but rather a combination of factors, the main one being that after each word the listening process is restarted and any extra words or parts that might have been recognized before are erased. So for instance in a sequence like 'na mitsu' it might send only 'na' (because of a requirement that words are stable for 300ms) and then 'mitsu' is skipped.

I tried to fix the issue by removing the restart but I wasn't able to make it work yet. So Ive switched and now I'm working on implementing the features requested in this issue.

JackMostow commented 6 years ago

@octavpo - Discussing how Listener works would have been much more timely months ago, and might already be too late.

The above description is frustratingly vague -- e.g. that MultiMatch is "pretty much disabled." A "real example" would be well-chosen to illustrate an issue and include:

Unless Listener has changed drastically since the Reading Tutor, you may (or may not) be confusing two requirements:

  1. A 300 ms silence threshold to terminate an utterance
  2. A shorter threshold to consider a word stable.

Also, it's possible that ASR output is supposed to be concatenated over multiple utterances as input to MultiMatch, rather than input to MultiMatch one utterance at a time to track the reader through the text.
Unfortunately I don't recall if we implemented the concatenative approach or merely contemplated it. I'm not sure if it's equivalent to what you called "removing the restart".

Now that you're back, we should meet ASAP to look at a real example and trace the code involved, but I won't be in till 9/26. For right now, please post a real "real example" as I requested on 9/1/2018, with a detailed explanation including links to the specific relevant locations in source files.

JackMostow commented 6 years ago

From: Octav Popescu octav@cmu.edu Sent: Thursday, September 27, 2018 6:39 PM

I have returned from vacation on 9/17, but I've returned to LA. I'm coming back to Pittsburgh on 10/16. The skipping aspect of issue #348 (parts 1 and 2) do have the effect to compensate the issue with dropping words, although maybe not in all cases. I've implemented it to happen in all modes that involve listening to students, not just ECHO mode, I don't see why it would be restricted to that. The aspect related specifically to ECHO mode (parts 3 and 4) probably won't have any influence. Octav

JackMostow commented 6 years ago

So are these the old -> new behavior changes of the modes that listen to students, where "sentence" means a text segment as long as a sentence or as short as one word?

octavpo commented 6 years ago

No, not sure why you think it would work like this. If it's related to my previous message, I was trying to say that the word skipping behavior described in the first two points under changes in the first message is common to all modes, not the new echo mode.

JackMostow commented 6 years ago

Thanks for clarifying. I misunderstood 1-4 as applying to all modes, not just 1-2 as you in fact said.

JackMostow commented 6 years ago

@octavpo (@kevindeland, @judithodili , @eyarzebinski , @amyogan take note) - I need your update by this Wednesday 10/3 for it to have any chance of getting into the upcoming beta test.

JackMostow commented 6 years ago

@octavpo - When will your update be ready? - Jack

octavpo commented 6 years ago

I'm trying to finish it but it's no easy. The skipping code works fine. The echo mode still has a few issues, not sure how long I need to finish it.

octavpo commented 6 years ago

Btw one issue I found and I can't solve on my own, probably familiar to you, is that the segmentation timing is not very accurate. In my testing it seems it cuts off the last word, so I added another 100ms after it. Which seems to be fine for the few examples I was running, but probably not a good general solution.

JackMostow commented 6 years ago

@octavpo - Can you TODAY (Wed. 10/3 your time) make a stable branch with just the skipping code, so we can try it ASAP?

@kevindeland needs all code by Thursday night 10/4 or Friday (I forget which) to prepare to deploy Monday.

Please point me to an APK. I assume you know how to generate them in general, but ask @kevindeland if you need RoboTutor-specific help (e.g. regarding assets). Please turn on skipping by default but include a switch (e.g. a configuration parameter) to disable it easily. Ask @ealanhill how to use his configuration parameter mechanism. Make sure it behaves reasonably for 1-word "sentences". Does it behave visibly as if accepting individual skipped words? That is, does it fix most false rejections? If so I want it even if you don't fix echo mode in time -- though I still hope that you will, in which case it too should have a switch. One lesson from Project LISTEN is that a feature is something you can turn off.

octavpo commented 6 years ago

It took a while to merge it with the current development branch, but I think it's fine now. The PR #386 includes the previous change about having different reading modes for successive sentences (which hadn't been merged into development for some reason), since they're in the same tutor.

Not sure about APKs, where do you guys put them? Not sure where I can find a tutor with 1-word sentence to test, I haven't tried that.

There is a switch in the code to enable/disable skipping, but not in configurations, I'll see about that. It does accept skipped words but leaves them black, as requested. Not sure if it fixes most false rejections, it can still happen that the next word is again wrong, and then it's rejected.

JackMostow commented 6 years ago

@octavpo - I wish you had just put the apk somewhere instead of incurring further delay by waiting for me to say where. Please put it in Downloads from any team member, and post a link to it here so that I'll get notified.

Any of the word reading activities have lists of individual words to read.

@kevindeland - Any requests/suggestions to Octav for changes to his test apk, e.g. where or how to include it under the activity matrix or CUSTOM menu in the debugger menu if it doesn't simply affect all story reading activities with modes that listen to the kid (PARROT, ECHO, READ, REVEAL)?

Thanks. - Jack

eyarzebinski commented 6 years ago

@octavpo story_4 as it appears in apk 2.3.0.1 (note that there was a story resort, so this might not be story_4 in old apks) has a 1 word sentence: first sentence: Anaitwa Bi Simu. second sentence: Mwangalie!

JackMostow commented 6 years ago

Where’s the apk to try out toleration of individual word skips?

octavpo commented 6 years ago

I just dropped it to the Downloads area.

JackMostow commented 6 years ago

@octavpo - I installed it (after uninstalling RoboTutor and having to restart the tablet) but it crashes on launch, without even installing assets. Did you try installing and launching it? What .zip files does it require or prohibit from being in the Download folder?

octavpo commented 6 years ago

I have tested my changes in version 2.0.0.1, which was the current development version when I started. My code doesn't need any extra resources, but I think version 2.3.0.1 in development does. When I tried to run that last night, it wasn't crashing but it was showing an empty screen (but the audio was still playing, so it wasn't crashed). So I have reloaded all resources from CodeDrop1_Assets and then it was starting fine. I've copied the zips to the Google drive downloads if you need them, I don't now which one is new.

Last night I didn't have time to actually run a story tutor. Now when I tried it didn't seem to recognize anything. However neither did the development branch (which doesn't include my changes). After some debugging I found out somebody had introduced a switch to disable the audio recording, and it was set to disabled. But that also means the voice recognition doesn't get any signal. So I reenabled it and created a new APK, which now seems to be working as expected. So you also need to reload it.

I also don't see any tutors with any modes other than HEAR or READ in the matrix, have you guys decided to abandon them? If so should I continue working on the ECHO mode rather than fixing the issue with dropped words?

JackMostow commented 6 years ago

The QA for beta has only new activities so it used only HEAR and READ but the full matrix will include others. If you need stories to test ECHO, can you just use existing ones? If you need help, ask Kevin.

JackMostow commented 6 years ago

When I tried to install the apk, it says there was a problem parsing it. When I try again, it crashes on launch, even if I move all the .zip files into a subfolder of Download to hide them, or all of them except Code drop 1 Master.

Your code needs to work with the latest Code drop 2 assets, not just Code drop 1.

Please try to make it work with the assets in QA Download Pack 09-26.

The install-launch-crash-fix-recompile cycle needs to happen and succeed at your end, right away -- it's WAY too slow if the install-launch-crash part happens at my end.

If we can't test your code, we can't use it.

octavpo commented 6 years ago

I tested it with the new zips and it still works for me. Here's my current list of zips:

CodeDrop1_LitAudio.1.1.0.zip CodeDrop1_LitStories.1.1.0.zip CodeDrop1_NumberStories.1.1.0.zip CodeDrop1_Songs.1.1.0.zip CodeDrop2_MathStories.1.1.0.zip CodeDrop2_ReadingStories.1.1.0.zip CodeDrop2_Tutors.1.1.0.zip RTAsset_Audio_SW_Set1.1.1.0.zip

Maybe I don't know how to create a proper apk, all I did was to copy the one Android Studio builds and stores in app/build/outputs/apk. If that only works with the Studio, sorry I don't know how to create another one.

JackMostow commented 6 years ago

@kevindeland - @octavpo's .apk crashes on launch before loading any .zips. Can you either point him to how to build a working .apk, or build one using his code for word-skipping and session audio recording? @octavpo - Is your ECHO code ready to include? Is it in a pull request? Can you tell Kevin which of your pull requests to include (if that's a thing) if he builds the apk?

octavpo commented 6 years ago

No the ECHO code is not ready yet, hopefully I can finish tomorrow. I managed however to fix the issue with the lost words (and sometimes parts of words) in the recognizer, now it works much smoother than before. The reading modes branch has all the relevant commits and nothing extra.

kevindeland commented 6 years ago

@octavpo if you're trying to generate an APK that other people can use, it needs to be signed. It sounds like you are using the APK that Android Studio builds automatically for debug?

These are the steps you'd have to follow: https://www.quora.com/How-do-I-build-an-APK-in-Android-studio

JackMostow commented 6 years ago

@kevindeland - Thanks for the pointer that I hope will enable @octavpo to create a usable APK right away. If he also needs a pointer to RoboTutor's key for creating a signed APK, please point him to it, and how to use it -- e.g. the password to use. Is it Authentication/android.jks in https://github.com/RoboTutorLLC/Authentication? @octavpo - Please confirm that you have read access to this repo and file.

JackMostow commented 6 years ago

@ealanhill - You asked at 11:04am 10/5/2018: Could you provide me with a link to Octav’s build? I’m not seeing the “XPRIZE - Admin” folder as shared with me.

Gladly! Did you already finish the other ~2 tasks that @kevindeland assigned you?

You probably don’t have access to the top-level folder XPRIZE – Admin, but you should have access to its subfolder Downloads from any team member, and within that subfolder to 2018-10-04-18.00-robotutor.debug.2.3.0.1.apk, owned by @octavpo and renamed by me to distinguish it from @kevindeland's latest apk.

Let us know as soon as you solve the problem.

@octavpo and @ealanhill - I want to test the new behavior ASAP, so please get a working apk first. Once that's done, it would be even better if you can:

Thanks! - Jack

ealanhill commented 6 years ago

@octavpo Looks like something went wrong with the build process, you're missing an important class in the APK. Here's the full stack trace:

2018-10-05 11:24:17.708 6887-6887/cmu.xprize.robotutor E/AndroidRuntime: FATAL EXCEPTION: main
    Process: cmu.xprize.robotutor, PID: 6887
    java.lang.RuntimeException: Unable to instantiate activity ComponentInfo{cmu.xprize.robotutor/cmu.xprize.robotutor.RoboTutor}: java.lang.ClassNotFoundException: Didn't find class "cmu.xprize.robotutor.RoboTutor" on path: DexPathList[[zip file "/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk"],nativeLibraryDirectories=[/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/lib/x86, /system/fake-libs, /data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk!/lib/x86, /system/lib, /vendor/lib]]
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2718)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892)
        at android.app.ActivityThread.-wrap11(Unknown Source:0)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1593)
        at android.os.Handler.dispatchMessage(Handler.java:105)
        at android.os.Looper.loop(Looper.java:164)
        at android.app.ActivityThread.main(ActivityThread.java:6541)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767)
     Caused by: java.lang.ClassNotFoundException: Didn't find class "cmu.xprize.robotutor.RoboTutor" on path: DexPathList[[zip file "/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk"],nativeLibraryDirectories=[/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/lib/x86, /system/fake-libs, /data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk!/lib/x86, /system/lib, /vendor/lib]]
        at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:93)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
        at android.app.Instrumentation.newActivity(Instrumentation.java:1173)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2708)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892) 
        at android.app.ActivityThread.-wrap11(Unknown Source:0) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1593) 
        at android.os.Handler.dispatchMessage(Handler.java:105) 
        at android.os.Looper.loop(Looper.java:164) 
        at android.app.ActivityThread.main(ActivityThread.java:6541) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767) 

In order to avoid this, please follow the directions found in the link Kevin provided (https://www.quora.com/How-do-I-build-an-APK-in-Android-studio).

If you have the lastet version of Android Studio installed, there are a few slight differences. Go to Build > Generate Signed Bundle / APK, select the APK radio option and click Next. From there follow the directions in the link to generate a signing key and the APK.

@kevindeland @JackMostow if there is a branch that @octavpo has been working off of to create this, I might be able to generate a quick APK if y'all provide the branch name/link.

JackMostow commented 6 years ago

@ealanhill - Judging from https://github.com/RoboTutorLLC/RoboTutor/branches/active: reading_modes updated 6 hours ago by @octavpo remote_configuration updated 3 days ago by @ealanhill

ealanhill commented 6 years ago

@JackMostow I uploaded a debug build to the same folder: https://drive.google.com/open?id=1ZUIHwtaGq44rrgrfF3loIcBX0PErcurS

It has all the classes, but I'm concerned it won't work because it's a debug version that is signed by my debug cert, so it may throw a fit when you install it on a device. If it doesn't, great!

The version code looks to be off from the one @octavpo created, but it is built from the latest on the reading_modes branch. Here's to hoping it works and provides you what you need to complete the testing.

JackMostow commented 6 years ago

Just saw your post. Will try immediately!

JackMostow commented 6 years ago

Good news: installs, opens, loads assets. @octav - Bad news: story.hear::story_54__it takes at least several seconds to load with no sign of progress, just a gray screen. It keeps repeating the sentence "Hapo zamani ... blanketi" but the first word stays underlined. story.parrot::story_4 also takes a long time to load and then keeps repeating the first sentence. story.echo::story_4 listens to first sentence but then keeps repeating it. story.parrot:: (parrot letter A) also takes a long time to load and then keeps repeating the first sentence.

Good news: not the fault of the apk building!

JackMostow commented 6 years ago

@ealanhill - Thanks for cracking the apk issue. Can you clarify "The version code looks to be off from the one @octavpo created"? Does it have anything to do with the version number being 2.0.0.0?

@octavpo - story.read::story_22 doesn't have the bug above, but it has as much trouble as ever recognizing my voice.

RoboTutor wrote the recording as RoboTutor_debug.2.0.0.0_2018.10.05.13.59.56_6105001158.wav, but it sounds severely sped-up.

ealanhill commented 6 years ago

@JackMostow yes, it says that it's version 2.0.0.0, and I think Robotutor is much higher than that

octavpo commented 6 years ago

The code on reading modes is 2.3.0.1, so I don't know what built you guys are using, but it's not from the current branch code.

I was trying to build an apk myself, but since @ealanhill was saying I should be on the latest version and I had ignored Studio's message to update for the last week, I thought it would be a good idea to do so. But then the update went wrong, so I had to reinstall it. That worked, but now when I try to build I get a message saying I need to select Android SDK. No idea how to do that. Any help? I'm an Android 3.2.

octavpo commented 6 years ago

@JackMostow Before trying to update Studio I had run story.hear::story_54 and story.read::story_22, and they both work fine. 'parrot' and 'echo' tutors are not available on the matrix generated by this branch, as they're not on development either. That shows again you're not running my code.

octavpo commented 6 years ago

Ok it looks like I needed to do some sync-ing, now it compiles. I'll build an apk soon.

JackMostow commented 6 years ago

@octavpo - 5pm today is the deadline to submit code to Kevin after our testing shows that it works properly. You were off-line for 8 hours. Meanwhile, @ealanhill was kind enough to find and fix the bug in your code and build an apk I could finally run. The audio it recorded is bad, and the ASR seemed worse than without your changes. Under the circumstances I decided we had to do without them. Your apk should use what 2.4 or Kevin's next APK will. No promises.

octavpo commented 6 years ago

What you're saying about is not true, @ealanhill hasn't fixed anything and gave you an apk that's not the current state of the reading modes branch. And up to now it wasn't my job to create apks, so no surprise i didn't know how to do it. I do need to sleep from time to time.

Anyway, I've built an apk with the code from the branch and put it in the download area if you still want to test it, as you wish. It's still 2.3 at this point, I'll work on merging the latest development code next. I don't have a problem if you don't want to include it, but the speech recognition is working definitely much better in my testing.

JackMostow commented 6 years ago

I’m recording Leonora till ~5 but will then try your apk. Are you sure Alan’s apk didn’t include your branch?

octavpo commented 6 years ago

I'm sure it wasn't the current version on the branch, since it was showing 2.0 instead of 2.3 and it had a different activity matrix.

I've rebased my branch to the latest version on development, 2.5.0.0, and put a new apk in the download area. I don't get an activity selector matrix since the merge though. I thought it was because I had chosen a release version, but now I chose debug and I still don't see it.

octavpo commented 6 years ago

It seems the missing activity selector matrix is because Kevin has just pushed the external configuration facility to the development branch. So the code now defaults to a release version unless you have a configuration file that says otherwise. So to get it back you need to push the config.json file I just put in the Google drive download area to /sdcard/Download.

JackMostow commented 6 years ago

octav_robotutor.release.2.3.0.1.apk had the same repeated-sentence problem and no debugger menu so I couldn't test ASR. I just installed octav_robotutor.debug.2.5.0.0.apk.