Image Classification Video Scavenger Hunt example doesn't work when p5.Speech doesn't work

[hx2A]

→ Description 📝

I found a bug/issue with the Image Classification Video Scavenger Hunt example:

https://ml5js.github.io/ml5-examples/p5js/ImageClassification/ImageClassification_VideoScavengerHunt/

The expected behavior is for it to repeatedly try to classify what it sees on my webcam and to say out loud what it sees.

This example works fine on my Chromebook with Chrome. It does not work correctly on my Linux machine using Chrome. There is no speech and it does not try to repeatedly classify what it sees.

I debugged the issue and figured out the problem. The core issue is that the p5.Speech() library does not work (for me) on Linux/Chrome and no speech is generated. The speech library also doesn't work using the example code from ability.nyu.edu. The speech library just doesn't work on this machine with Chrome.

I believe the reason why it doesn't work on my Linux machine is because myVoice.voices is an empty list. This is not the case for my Chromebook. This library is dependent on the browser to do the speech synthesis and to have "voices" available. For whatever reason, this machine doesn't supply any.

The speech issue is probably a software issue with my machine and is not the subject of this bug. However, this shortcoming undermines the ml5 example because the onEnd method is never triggered, and that's what this bug is about.

  // speechEnded function will be called when an utterance is finished
  // Read more at p5.speech's onEnd property: http://ability.nyu.edu/p5.js-speech/
  myVoice.onEnd = speechEnded;

Because the p5.speech library doesn't say anything, the onEnd method is never called and the speechEnded function is never called.

function speechEnded() {
  if (isPlaying) classifyVideo();
}

That means classifyVideo() is never called again. Therefore, this example tries to classify what it sees once and then stops. The end result is confusing and makes the example seem broken.

I propose that the example code be modified to check myVoice.voices to see if voices are available. It should only attempt to speak if voices are available. If not, it should display a reasonable message for the user informing them that speech won't work but continue the ML demo by displaying messages to the #message field only.

I'm happy to make this bug fix but wanted feedback from others first.

Version 74.0.3729.169 (Official Build) Built on Ubuntu , running on Ubuntu 19.04 (64-bit)

[joeyklee]

@hx2A - Wow! Thanks so much for the excellent investigation. It is super helpful that you run this on different operating systems and managed to diagnose this issue.

Perhaps one thing we need to add Browserstack tests for trying to handle these kinds of moments.

I agree that it is a good idea to add an error handling here. If you'd like to make a PR, that would be great. I think @shiffman can also review for levels of "friendliness" and we can go from there. Does that sound ok?

Thanks so much for this detective work! 🙏

[hx2A]

You are welcome! I checked using my Chromebook because I was suspected that the problem I was experiencing was specific to this machine. It happens the Chrome screenreader doesn't work either.

I like the idea of using Browserstack tests but think it should be a part of a broader effort to improve the unit tests. Right now some of the tests fail randomly or fail consistently, which is a bad thing. That teaches people to ignore unit test results, which in turn teaches people that unit tests aren't useful and there is no point in writing more of them. One of my goals is to change that. To get the unit tests working correctly, and to teach people that unit tests are useful.

Sure, I'll make a PR for this. I'll take care of it this week.

[joeyklee]

@hx2A - you're totally right about building trust in the tests and software. I know @brondle also has an interest to get all this up to speed as well, so you're definitely in good company! Thanks for your nice reflections here!