IBM / taxinomitis

Source code for Machine Learning for Kids site
https://machinelearningforkids.co.uk
Apache License 2.0
143 stars 137 forks source link

Move creation of image training zips to serverless #210

Closed dalelane closed 5 years ago

dalelane commented 5 years ago

The most computationally expensive thing in the app is what happens when a student clicks the "Train new machine learning model" button for image projects.

This has to:

The main bit of that starts here: https://github.com/IBM/taxinomitis/blob/85b3d1c59a8580b54875d62cc7d5c844dd0bd27b/src/lib/utils/downloadAndZip.ts#L329

That is memory intensive, so the app is provisioned with more memory than it really needs to cope when a class of students all start training image projects at the same time. https://github.com/IBM/taxinomitis/blob/85b3d1c59a8580b54875d62cc7d5c844dd0bd27b/manifest-template.yml#L3-L4

It means the app runs at single-digit CPU percentage most of the time, and very low level memory and disk usage.

Offloading this task to a serverless function would mean I could hugely shrunk the deployment footprint for the main server.

dalelane commented 5 years ago

First version implemented.

I've described some background about how I've done it in https://dalelane.co.uk/blog/?p=3808