Testing the new TensorFlow integration for App Inventor

dalelane commented 3 years ago

Background

TLDR: I'm rewriting some of the App Inventor integration in ML for Kids and would like help testing it

Image projects on Machine Learning for Kids were originally based on IBM's cloud service "Visual Recognition".

Whether students made their project in Scratch, Python, or App Inventor, all of these integrations were actually sending their requests to train models and classify images to the Visual Recognition service (via ML for Kids).

The Visual Recognition service is being withdrawn in December 2021, so I need a different approach that removes the dependency on this.

I've already fixed this for projects that students make in Scratch and Python - those integrations now do their own training and classifying, using the TensorFlow library.

I'm working on doing the same for App Inventor.

It's been a while since I've done any Android development, and the App Inventor development environment is... a little unfriendly. So this needs more testing than I can do by myself.

I'm looking for kind volunteers to try out the ML for Kids integration with App Inventor on their Android phones.

What's next?

It's not ready just yet - I'm planning ahead. But I think I'm only a week or two away from having something working.

I'll give you step-by-step instructions to follow
- Don't worry if you've never used Machine Learning for Kids and/or App Inventor before!
It won't take long
- e.g. I'll give you a sample ML project to start from, so you don't have to collect your own ML training example images if you don't want to. I'll make the whole thing as quick as possible

I'll ask you to tell me:

if it works at all
anything else you notice
- (e.g. does it take long to train? does it take long to classify?)

If you're familiar with adb, I'll also ask if you'd mind sending me the output from "adb logcat" while running it. No worries if you're not though. That's just a nice-to-have bonus.

gabrielcbe commented 3 years ago

Hey @dalelane, i'd be happy to help. Just let me know here or via email and i'll follow the steps to test it on my android phone and report it back.

dalelane commented 3 years ago

I think I have something ready for testers!

I've set up two options, depending on how much time you have and how familiar you are with App Inventor.

Very quick testing

I've used the App Inventor ML4K integration to create a quick-and-simple Android app.

Please download the app apk and install it on an Android device.

It is hard-coded to use an existing machine learning project, which has a (very small!) training data set of photos of cats and dogs.

Please train a model (by clicking the Train Model button to create an ML model using that pre-defined training set)

Use it to classify photos (that you take of cats or dogs using your phone's camera) If you don't have a cat or dog, a photo of another screen with a photo should work

(My focus is on whether the ML model trains on your device at all, and whether it can be used to classify photos from your camera at all. The actual ML accuracy isn't a priority for this, as the training set isn't great.)

In-depth testing (if you know App Inventor)

If you've used App Inventor before with Machine Learning for Kids and know how the extension works, you can download a modified version of the extension and use it to create a custom project.

You'll need to create an ML4K project for training images on your computer (not in the cloud), get an ML4K API key from the Python page, and then use it as normal.

Please let me know if it works!

adamcohenrose commented 3 years ago

Hi Dale,

My first quick attempt - installing on a Pixel 3 running Android 11: I got a big error saying that the WRITE_EXTERNAL_STORAGE permission had not been enabled… The APK says it will only request WRITE_EXTERNAL_STORAGE up to SDK 29 (Android 11 is level 30).

The ML model trained successfully, but I couldn't take a photo -- pressing the button resulted in the same error coming up again.

I'm guessing this could be because the app has requested Media access for the pictures, but is trying to save them in a fixed location rather than Context.getExternalFilesDir(null) or Context.getExternalCacheDir(), or else is trying to save to a named directory in Context.getExternalFilesDir(String) rather than using the default (apparently disallowed in Android 11 or above).

When I installed on an Android 8.1 device, I didn't get the big error and the ML model also trained. However, tapping on the Take photo button did nothing at all :-(

logcat had the following, which also seems to indicate some permission problems:

2021-09-14 08:40:29.053 5132-5132/? D/NougatUtil: packageName = appinventor.ai_dale_lane.tfjsbeta
2021-09-14 08:40:29.056 1328-3405/? I/ActivityManager: START u0 {act=android.media.action.IMAGE_CAPTURE flg=0x3 cmp=org.codeaurora.snapcam/com.android.camera.PermissionsActivity clip={text/uri-list U:content://appinventor.ai_dale_lane.tfjsbeta.provider/external_files/Pictures/app_inventor_1631605229053.jpg} (has extras)} from uid 10054
2021-09-14 08:40:29.056 483-4657/? D/msm8916_platform: platform_set_channel_map: set mapping(1 2 0 0 0 0 0 0) for channel:2
2021-09-14 08:40:29.056 1328-3405/? W/ActivityManager: Appop Denial: starting Intent { act=android.media.action.IMAGE_CAPTURE flg=0x3 cmp=org.codeaurora.snapcam/com.android.camera.PermissionsActivity clip={text/uri-list U:content://appinventor.ai_dale_lane.tfjsbeta.provider/external_files/Pictures/app_inventor_1631605229053.jpg} (has extras) } from ProcessRecord{8ccbe67 5132:appinventor.ai_dale_lane.tfjsbeta/u0a54} (pid=5132, uid=10054) requires android:camera
2021-09-14 08:40:29.057 5132-5132/? I/Form: Form Screen1 got onPause
2021-09-14 08:40:29.058 5132-5132/? I/Form: Form Screen1 got onActivityResult, requestCode = 2, resultCode = 0
2021-09-14 08:40:29.058 5132-5132/? I/CameraComponent: Returning result. Request code = 2, result code = 0
2021-09-14 08:40:29.058 5132-5132/? I/CameraComponent: Could not delete file file:///storage/emulated/0/Pictures/app_inventor_1631605229053.jpg

dalelane commented 3 years ago

@adamcohenrose Thanks very much for this.

I'm new to both Android and App Inventor development, so I'm open to suggestions if you've got any ideas how to fix this.

For the storage permissions in particular:

I tried requesting permissions in the App Inventor project: (I'll try adding Camera to this list to fix that issue)

And in the extension source code: https://github.com/kylecorry31/ML4K-AI-Extension/blob/8ecadaad717582cd0207bb782a6eb0ae6e08c74f/src/com/kylecorry/ml4k/ML4KComponent.java#L49

so I'm not sure what else to try.

Any ideas?

dalelane commented 3 years ago

@adamcohenrose I've updated the apk link above to https://www.dropbox.com/t/8YS8YURvZlkyzbtW which has been updated to use this App Inventor code

dalelane commented 3 years ago

In case people have problems with the apk that can be explained by me doing something silly with App Inventor (entirely likely, as I'm still new to App Inventor), here is what I did:

adamcohenrose commented 3 years ago

I think WRITE_EXTERNAL_STORAGE is no longer valid for Android 11 -- more details in that AppInventor thread

I've tried creating a project with the Pro Camera extension, but it's not working yet... It does seem to take a picture, but isn't making the file available to the ML code.

I haven't tried the Legacy option yet -- might be the simplest approach given that projects are unlikely to be sent to the Play Store initially...

ml4kidstest.aia.zip

dalelane commented 3 years ago

@adamcohenrose That forum thread was a huge help, thanks.

I've tried enabling the legacy file mode - latest apk with that turned on is https://www.dropbox.com/t/sWm51aMA9qFd2b7G

IBM / taxinomitis