Support for image recognition in App Inventor

[ruizrube]

Dear Dale. First of all, thanks for your great tool. We are using it in a training course on Artificial Intelligence with App Inventor for high school teachers from all over Spain. Now that IBM Watson has deprecated its API for image recognition, I wonder if it is expected for ML4K to provide support for those models created in the browser for App Inventor. If so, when is planned for?

Best Regards

[dalelane]

First up - I should say - I'm sorry for the disruption. I really didn't see the deprecation coming, and I'm very sorry for any problems this has caused.

Secondly, (and more aimed at anyone else who comes across this thread and wants to know the context) the current situation can be found in the FAQs at https://machinelearningforkids.co.uk/help (specifically, the "Which students can train images machine learning projects in the cloud?" question, which I'll keep up to date to identify who can still use the VR service with App Inventor).

To your specific question:

I wonder if it is expected for ML4K to provide support for those models created in the browser for App Inventor. If so, when is planned for?

If someone wanted to contribute a Java implementation of an image classifier (e.g. some sort of TensorFlow Lite implementation of transfer learning in Java that would run on Android) that would be fantastic - I'd be eternally grateful and very happily accept that.

I don't expect to have time to work on this myself in the immediate/short term - certainly not in the next couple of months.

I realise that I'm not explicitly answering your question, and that it would make your planning easier if I could give you a definitive "I will implement this feature by this date" plan - but I'm afraid I can't make that sort of commitment. I don't have time to work on it at this time, and I can't really promise when that'll change.

[ruizrube]

Thank you for your prompt answer. It was difficult to see that deprecation and I perfectly understand you cannot make the commitment to solve that.

Recently, App Inventor team has released an extension [1] called Personal Image Classifier [2], which can load an ML model created with a specific webapp for training purposes [3] (I guess based on TensorFlow). But I don't know if that extension could be easy compatible with the models created with ML4K.

[1] https://mit-cml.github.io/extensions/ [2] https://appinventor.mit.edu/explore/resources/ai/personal-image-classifier [3] https://classifier.appinventor.mit.edu/

Thanks.

[dalelane]

Thanks very much for the links - I wasn't aware of those, and that is super helpful. It does suggest options for quicker/easier ways forward.

At first glance, the model side of things (exporting an ML4K model in a format that could be consumed by their AIX extension) looks like it should be reasonably straightforward. I'm sure there will be unexpected complications I haven't thought of yet, but in principle it feels like it should be possible for ML4K to offer TensorFlow mobilenet models for download in a zip, which it looks like https://classifier.appinventor.mit.edu/ is doing.

That would then mean either just adopting the App Inventor aix extension as-is, or making something that behaves similarly to it from a ML side of things but providing blocks that are closer to what https://github.com/kylecorry31/ML4K-AI-Extension has now.

@kylecorry31 @MrMazzone What do you think?

[kylecorry31]

@dalelane Joe and I talked about this earlier this week and wanted to know if it would be possible for you to implement training of a model on the server-side, which would then mean the only change to the ML4K AIS extension would be a URL swap.

If that isn't feasible, I believe the best course of action here would be to convert the ML4K model into a format for consumption by the Personal Image Classifier AIX extension (and potentially a new extension in the future). Unfortunately, I don't foresee having any time to re-implement an AIS extension with support for on-device transfer learning.

[dalelane]

thanks very much, @kylecorry31

Unfortunately I don't think I could replace it server-side. It's not about the implementation, so much as I think I probably can't afford to host it. Hosting the training for ML models in a scalable way can get prohibitively expensive - I suspect (although this is just my speculation - I have no insider knowledge or insight) that's why IBM isn't offering Visual Recognition as a standalone cloud service any more. I was already getting complaints about the time that training requests were being queued up was getting longer and longer, which is why I started looking into doing it in the browser.

Anyway... idle speculation aside...

I'm just looking now to see how the App Inventor extension was done - they're not doing it in Java, which I'd assumed they'd done, after all. They're doing it in JavaScript using TensorFlow.js the same way I'm doing it for Scratch, which they wrap in an HTML page, and then open in the App Inventor extension using a WebView.

Kind of sneaky, but - assuming it all works! - actually very neat. And much simpler than trying to do it all using Java APIs. It means we could even reuse the JavaScript implementation I've already done for Scratch.

I totally understand you not having time to do even this sort of simpler approach though - I similarly am finding it really hard to juggle day-job and family commitments, let alone extra stuff like this!

[kylecorry31]

@dalelane I figured that was the likely issue there - I imagine it must be pretty expensive to host something like that. Thanks for confirming!

That's actually a super smart idea. I know that when I built the ML4K extension, I could not figure out how to import external libraries and struggled with the outdated SDK version App Inventor was compiling with at the time.

If it helps, I actually did build an active transfer learning project in TensorFlow.js a couple of years ago: https://github.com/kylecorry31/DoodleNet/blob/master/public/js/nn.js

I don't know how easy it would be to port that into the extension the same way they did it - I can't make any guarantees but that does reduce the complexity a lot to the point it may be feasible to do. Though, I probably won't have time for several months.

[dalelane]

I know there are several people following this issue, so I'll put a quick status update here.

I finally started working on this issue over this past weekend (the work is happening in https://github.com/kylecorry31/ML4K-AI-Extension/pull/37 ).

It's going very well so far, and I haven't hit any significant obstacles. I don't think it should take very much longer to complete. It's difficult to put an exact date on this as I don't know how much time I will have to work on it, but I don't think it'll be more than a few weeks - and certainly I expect it to be comfortably do-able before the IBM Visual Recognition service is withdrawn in December. (At that point, App Inventor support for image projects in ML4K will no longer be possible without the work described in this issue - so that is very much the deadline I'm working to).

[ruizrube]

great. Thanks for your work.

[dalelane]

Another quick status update.

I have a beta version of the App Inventor extension working that supports image recognition using TensorFlow.js - which would let me re-enable App Inventor for all projects, whether they were trained with IBM Visual Recognition or not.

It needs a lot of testing before I can add it to the site - I'm tracking the testing separately in https://github.com/IBM/taxinomitis/issues/425

If I can get some verification that it works okay, I'll add it to the main ML4K site.