Open anssiko opened 4 years ago
@irealva to comment in this issue experiences of in-browser training with Teachable Machine, a project successfully used in cross-disciplinary contexts such as education, arts, for in-browser training to solve real-world problems. What are the limitations you facing and how you've worked around them? Size limits of the models, different browser behavior? Your input is valuable in determining gaps in web platform capabilities to be improved.
@nsthorat to comment on key learnings from designing TensorFlow.js for training models. My understanding is Node.js-based backend is primarily used for training, and that is mainly due to the following advantages over browser-based:
On the browser side, it seems you've managed to work around the limitations and the complexities of WebGL with a layered approach. As you're aware, WebNN API is to solve part of the issues, as well as WebAssembly with SIMD, but I'm curious whether there are other API gaps we've overlooked that we could look into addressing with standards-based Web APIs? Specific requirements with respect to filesystem access? Better memory management (discussed in https://github.com/w3c/machine-learning-workshop/issues/63)?
Teachable Machine features in-browser training for all the models in the app: the image, audio, and pose model. Users add their samples to different classes and train their own custom models directly in the browser. None of the samples ever hit our servers during training.
We get around some of these limitations by restricting the training to transfer learning, so the training is much faster. For example, in the case of the image model used in Teachable Machine, users just have to re-train 2 dense layers over their own data. All of the training code can be found in this repo if you want to take a look.
A few more technical notes:
Notes on usage:
@hapticdata might have some more thoughts.
As discussed yesterday here are a few additional considerations:
@cynthia's Machine Learning in Web Architecture talk makes a (wild) guess:
The current in-browser efforts (e.g. those pursued by the W3C's Machine Learning for the Web Community Group) are focused on inference rather than training. This is due to pragmatic reasons: limited availability of respective platform APIs to make this process efficient, also resource restrictions of the browser architecture not optimized for such a demanding task.
For in-browser inference, a model with a total weight size somewhere in the ballpark of ~100 MB starts to be too slow on a typical desktop hardware. For training, more memory and compute is required, so possibly even smaller models than that will be too slow to train in a browser to be useful in most use cases. @huningxin has probably couple of pointers to model size vs performance evaluations.
My questions:
For example, the large amounts of data needed to train a model is currently better ingested outside the browser from a native file system. The Native File System API may address the issue of data ingestion for in-browser usage. What other such API gaps would make the memory and compute intensive task of training more feasible in the browser context?