Teachable Machine Extension doesn't work in Main branch

[ctphillips]

Trying to use Extension for Teachable Machine with Micro:bit. Oddly, the Micro:bit extension (along with Scratch Link 2.0.2) works fine in the main endpoint (https://playground.raise.mit.edu/main/) but does not work (will not recognize any micro:bits) in the create endpoint (https://playground.raise.mit.edu/create/) same PC, same micro:bit. And the Teachable extension that is provided with the main endpoint, does not provide model outputs (the model loads, but when you provide say the audio that the model expects, the output doesn't update, even the model prediction variable shows no value, not even the background noise output, the variable simply shows nothing), however, the Teachable extension that is provided with the create endpoint works as expected. So I cannot connect Teachable Machine extension to the a micro:bit because in main, I can connect to the micro:bit but the Teachable model does not function, and in create, the Teachable extension works, but I cannot connect to a micro:bit. Not sure where to go from here, so reaching out to see if someone can assist.

[pmalacho-mit]

Thanks for your report, @ctphillips and for trying out both endpoints -- incredibly helpful!

For the microbit, are you using Scratch Link? If so, then that issue with the "create" branch is at least expected (if not ideal). There were some changes made to Scratch Link made by the Scratch team after our deployment of the "create" branch which caused some breaking changes -- we also attempt to preserve the "create" branch with minimal changes, which is why it remains broken.

However, those issues with the teachable machine extension on the main branch is something we definitely want to fix!

@mayarajan3 could you take a peek at this? My suspicion is that things have only broken for audio models (I at least can get image models working consistently once turning video on, let me know if you see any different @ctphillips ).

I also suspect when updating this extension, we opted to not have the microphone on by default (as it at least seems in the main branch we no longer turn the video on by default, but we do in the create branch). This is definitely an issue since there is no block to control whether or not the microphone is recording.

[ctphillips]

Yes, definitely using Scratch Link, so at least it is no surprise to y'all that connecting to the Micro:bit via Scratch link isn't working in the create branch.

Definitely, using an Audio Project on the Teachable Machine. I made sure the microphone was connected and unmuted on the page. I tried both Chrome, Edge and on my mac I tried Safari. All with the same issue. It is important to note when the audio model loads (paste the link into "use model" block and click on it) in the create branch if you are showing the model prediction variable on the Scratch stage, you immediately see the "noise background" value get set in the variable. That same sequence does not produce the expected noise background value in the model prediction variable (no value is set), which should happen even if the microphone is muted, since it is the default output of the model.

One other small observation. When you click on the Add Extension icon the Teachable Machine extension the choices look different between the main endpoint and create endpoint. The graphic for each are a bit different, and even the descriptions of the extension are different (of course they are both titled Teachable Machine, and the extensions themselves show the same block choices) but the Teachable Machine extension description in the main endpoint says "Use your Teachable Machine models in your Scratch project", while for the create endpoint the description says "Use Google Teachable Machine models in your Scratch project". It makes me think they are loading different versions of the Teachable Machine extension, but I don't know how to verify that.

Look forward to hearing what y'all figure out. I would love to use my audio Teachable Machine with my micro:bit in Scratch. Thanks for all the work that has been put into thus far!

-Craig

On Mon, Jun 24, 2024 at 1:15 PM Parker Malachowsky @.***> wrote:

Thanks for your report, @ctphillips https://github.com/ctphillips and for trying out both endpoints -- incredibly helpful!

For the microbit, are you using Scratch Link? If so, then that issue with the "create" branch is at least expected (if not ideal). There were some changes made to Scratch Link after the deployment of the "create" branch which caused some breaking changes -- we also attempt to preserve the "create" branch with minimal changes, which is why it remains broken.

However, those issues with the teachable machine extension on the main branch is something we definitely want to fix!

@mayarajan3 https://github.com/mayarajan3 could you take a peek at this? My suspicion is that things have only broken for audio models (I at least can get image models working consistently once turning video on, let me know if you see any different @ctphillips https://github.com/ctphillips ).

I also suspect when updating this extension, we opted to not have the microphone on by default (as it at least seems in the main branch we no longer turn the video on by default, but we do in the create branch). This is definitely an issue since there is no block to control whether or not the microphone is recording.

— Reply to this email directly, view it on GitHub https://github.com/mitmedialab/prg-extension-boilerplate/issues/368#issuecomment-2187143826, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABV6AFMLS5LK7VTOP45UPJ3ZJBO27AVCNFSM6AAAAABJ2EPJRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGE2DGOBSGY . You are receiving this because you were mentioned.Message ID: @.***>

[pmalacho-mit]

Hey @ctphillips -- all sounds good, and you are absolutely correct -- the teachable machine extension in the main endpoint is a later version than the one in create.

Thanks to @mayarajan3 's quick work (you rock, @mayarajan3 ), we deployed the likely fix: https://playground.raise.mit.edu/teachable-machine-audio-fix/

Want to make sure it works for your project and then we'll deploy the fix to main?

[ctphillips]

Thank you @mayarajan3 for putting together a fix! I verified that the fix endpoint does allow me to use a teachable machine audio model and connect to the micro:bit via Scratch Link.

One of the things I was trying to accomplish in this project was to use audio commands (via the teachable machine model) to have the micro:bit display icons on its LED array (which I can do using the micro:bit extension) and drive a couple of servos (I am using motor:bit board from Elecfreaks) which I cannot do, as there a no pin/servo blocks available in the micro:bit Scratch extension that I see. Is this something that could be added to the PRG project? I am looking at other ways to connect Scratch and micro:bit and get servo control, but I haven't seen anything slightly straightforward yet.

Thanks!

-Craig

On Wed, Jun 26, 2024 at 11:48 AM Parker Malachowsky < @.***> wrote:

Hey @ctphillips https://github.com/ctphillips -- all sounds good, and you are absolutely correct -- the teachable machine extension in the main endpoint is a later version than the one in create.

Thanks to @mayarajan3 https://github.com/mayarajan3 's quick work (you rock, @mayarajan3 https://github.com/mayarajan3 ), we deployed the likely fix: https://playground.raise.mit.edu/teachable-machine-audio-fix/

Want to make sure it works for your project and then we'll deploy the fix to main?

— Reply to this email directly, view it on GitHub https://github.com/mitmedialab/prg-extension-boilerplate/issues/368#issuecomment-2192190482, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABV6AFNJ26YNRNYDNEMVUW3ZJLWE3AVCNFSM6AAAAABJ2EPJRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGE4TANBYGI . You are receiving this because you were mentioned.Message ID: @.***>