Open mishushakov opened 3 years ago
Very cool! I'll need to read up more on how that would work, but the fact that it loads is a good start. Also the colab notebook seems to work fairly well for training with the split_data param I added, need to test that more though.
Sharing some great progress on the browser runtime:
Loaded the models (and weights) into a webpage
Modified predict.py
to save the X_ordered tensor as json file (for explanation see #5)
https://github.com/GuitarML/GuitarLSTM/blob/3c1dc9e379241c0f43369f2e60e88ffa0653df94/predict.py#L58
Note: due to big file size i was only able to export one second of the audio (65 megabytes)
Loaded the tensor into the TensorFlow.JS and was able to run the prediction first one is the input tensor and the second one is the output tensor
Loaded the resulting float32 data using Audacity's "Import Raw" function
The prediction run slowly on my 13-inch MacBook Pro (iGPU), but i will make a test on a PC with GPU i'd love to share the code once i figure out how to save the bytes as wav, i tried just writing them into a wav file, but didn't work
In the zip file you'll find the in.wav
and predict.wav
(again, generated through a browser)
actually spent whole night getting that to work, but totally worth it
Great stuff, yeah once you get on a roll it's hard to stop, get some sleep!
Kept my promise and made the web browser version possible try out yourself: https://mishushakov.github.com/GuitarLSTM-browser
get the example tensor here (it's so big i had to use git-lfs): https://github.com/mishushakov/GuitarLSTM-browser/raw/master/samples/tensor.json
if you're interested how it all works, feel free to fork the repo: https://github.com/mishushakov/GuitarLSTM-browser
thank you everyone for your help in making this possible to me this is just the beginning
😃
so cool, even works on my iPhone lol
@mishushakov Can't wait to try it out! Nice work!
on the training side it is possible to convert the the js models to keras (h5) this means we can train models in the browser and load them into plugin
tensorflowjs_converter --input_format=tfjs_layers_model --output_format=keras models/ts9/model.json models/ts9/ts9.h5
taken from https://github.com/tensorflow/tfjs/tree/master/tfjs-converter#javascript-to-python
I think so, but for the plugin I was planning on using json format for the models anyway. H5 is just another data format, as long as all the weights are in the json file we can make it work. And we can work together to make sure the models trained from the browser can be loaded into the future plugin.
H5 is nice because it compresses the data, but I think json is the better option because it's readable in any text editor, and I'm more familiar with loading json in c++.
@GuitarML not sure if it's a good idea give SavedModel a try
once you have that, you can load the model in every version of tensorflow (you read it right, tensorflow, not just keras) in C++ specifically you can use the tensorflow C++ API or tensorflow Lite C++ API
if this is still isn't enough you could use tensorflow-onnx to convert your model to onnx and load it with onnx runtime
i've been actually wondering whether we can make both networks run in your plugin one solution would be adding wavenet to the tensorflow version the other solution would be converting both to onnx
Hey there, i was able to load the model successfully in browser using tensorflow.js
to convert the models you can use TensorFlow.js converter
this issue is a backlog for inferencing and (possibly) training models in browsers