Allow for offline ml5.js & allow for all models to be served locally

joeyklee commented 5 years ago

→ Description 📝

want to suggest an update 💡
have an idea for a new feature 🆕

Hi! We should allow for ml5 to run without offline and have a backup of the models as part of our versioning

nsthorat commented 5 years ago

This is a great idea. FWIW you can use model.save() in tfjs land to put things in local storage or IndexedDB.

joeyklee commented 5 years ago

@nsthorat - ooooh yes. Thanks for this tip! We definitely should keep this in mind also for the generic neural net class: https://github.com/ml5js/ml5-library/pull/485

Local storage is awesome, but for our less-familiar users might be confused about where the data are in the end.

Just stream of consciousness here and maybe this is getting real crazy, but in the same way that there's tfhub or the upcoming Teachable Machine cloud space, it would be rad to have a web space/place to easily host and access these models. Something like models.ml5js.org or playground.ml5js.org.

I wonder too if we did a simple ml5-model-sandbox where you could "just" drop in your models into a folder on your local machine and have a little local server that serves models locally at localhost:{port}. Then for our more enterprising users, they can deploy that thing to some platform as a service to fit their needs and to separate out where their models are living vs. their app code. In the web/geo world, this is done with serving up basemap "tiles" which seems to be working for better or worse.

I'll definitely put some further thoughts here. Thanks!

joeyklee commented 5 years ago

@brondle - Hi! I hope you're well. In case you're interested, I started this repo -- https://github.com/ml5js/ml5-data-and-models-server -- to get rolling on allowing ml5 to run offline.

I figure we can start by:

creating a script that allow us to download all (or specified) of the models locally. Something like npm run fetch --all or npm run fetch --bodypix
then serve those models at using: npm start
we can integrate into ml5 a method or mode to indicate something like local: true

We can take some inspiration from @oveddan here: https://github.com/oveddan/posenet-for-installations

Any thoughts or contributions are welcome! Thanks!

hobg0blin commented 5 years ago

@joeyklee i'm down to hop on this!

joeyklee commented 5 years ago

@brondle - Awesome! Happy to also meet up in NYC/ITP if there's interest to co-work. Otherwise, via github/slack is great too!

shiffman commented 5 years ago

This is a great idea! For teaching and some total beginner contexts (like high school workshop or ICM-level at ITP) it might be also useful to support just pass in a local path to a model. This is related to elated to #379. For example:

User downloads MobileNet model and puts it in a folder called mobilenet_model in their sketch folder:

sketch.js
index.html
mobilenet_model/___model_files____

// Thens something like the following:
const classifier = ml5.imageClassifier('mobilenet_model/model.json', modelReady);

I guess the question is. . is it easy to just download the model files for many of the models we're using?

joeyklee commented 5 years ago

@shiffman - Thanks for the feedback.

Often these models come with a bunch of files, so it's not super straightforward to download them. I think the goal of having this tool could then be two-fold:

allow ml5 to run offline by providing more advanced users to use some scripts we write to download and serve those models from ml5-data-and-models-server
Allow us to create a simple web interface to make it easy for people to download the specific model they are interested in and as you suggested, to put them in the local folder they are working from.

joeyklee commented 5 years ago

Hi! Just a quick note/updates on this front.

Model and data server up and running!

I've create a repo - https://github.com/ml5js/ml5-data-and-models-server - that allows us to download all of the models and their associated weight files from across the web (most are on google storage, but some live elsewhere) and serve them up with a simple express server.

Feel free to give this a whirl!

Next major todo

The next thing we should figure out is how to handle redirecting the requests to the models in ml5.js. In a number of the functions like DCGAN and CVAE we point to a local manifest.json file living within the same directory as the ml5 project. This is similar to what @shiffman is suggesting above.

Any thoughts on this would be great cc @shiffman @yining1023 @oveddan

Feedback wanted

There's one major question I posed about directory structure for the offline models here: https://github.com/ml5js/ml5-data-and-models-server/issues/3 - any thoughts on this would be appreciated.

joeyklee commented 4 years ago

Just making a note for those following this thread:

I've added offline ml5 model support for most ml5 features now with #553
You can download all or selected models using the super beta version of this: https://github.com/ml5js/ml5-data-and-models-server

joeyklee commented 4 years ago

This should be resolved with #553. I will close this for now, but we should definitely consider adding the following:

Add saving models to local storage or indexdb

ml5js / ml5-library