GuitarML / GuitarLSTM

Deep learning models for guitar amp/pedal emulation using LSTM with Keras.
https://guitarml.com/
GNU General Public License v3.0
373 stars 48 forks source link

Browser support (discussion) #3

Open mishushakov opened 3 years ago

mishushakov commented 3 years ago

Hey there, i was able to load the model successfully in browser using tensorflow.js

Screenshot 2020-12-06 at 15 18 29

to convert the models you can use TensorFlow.js converter

pip install tensorflowjs
tensorflowjs_converter --input_format keras models/ts9x/ts9.h5 models/ts9/lite

this issue is a backlog for inferencing and (possibly) training models in browsers

GuitarML commented 3 years ago

Very cool! I'll need to read up more on how that would work, but the fact that it loads is a good start. Also the colab notebook seems to work fairly well for training with the split_data param I added, need to test that more though.

mishushakov commented 3 years ago

Sharing some great progress on the browser runtime:

  1. Loaded the models (and weights) into a webpage

  2. Modified predict.py to save the X_ordered tensor as json file (for explanation see #5) https://github.com/GuitarML/GuitarLSTM/blob/3c1dc9e379241c0f43369f2e60e88ffa0653df94/predict.py#L58 Note: due to big file size i was only able to export one second of the audio (65 megabytes)

  3. Loaded the tensor into the TensorFlow.JS and was able to run the prediction first one is the input tensor and the second one is the output tensor

    Screenshot 2020-12-08 at 04 55 29
  4. Loaded the resulting float32 data using Audacity's "Import Raw" function

    Screenshot 2020-12-08 at 06 06 48

The prediction run slowly on my 13-inch MacBook Pro (iGPU), but i will make a test on a PC with GPU i'd love to share the code once i figure out how to save the bytes as wav, i tried just writing them into a wav file, but didn't work

In the zip file you'll find the in.wav and predict.wav (again, generated through a browser)

Demo.zip

mishushakov commented 3 years ago

actually spent whole night getting that to work, but totally worth it

GuitarML commented 3 years ago

Great stuff, yeah once you get on a roll it's hard to stop, get some sleep!

mishushakov commented 3 years ago

Kept my promise and made the web browser version possible try out yourself: https://mishushakov.github.com/GuitarLSTM-browser

get the example tensor here (it's so big i had to use git-lfs): https://github.com/mishushakov/GuitarLSTM-browser/raw/master/samples/tensor.json

if you're interested how it all works, feel free to fork the repo: https://github.com/mishushakov/GuitarLSTM-browser

thank you everyone for your help in making this possible to me this is just the beginning

😃

mishushakov commented 3 years ago

so cool, even works on my iPhone lol

GuitarML commented 3 years ago

@mishushakov Can't wait to try it out! Nice work!

mishushakov commented 3 years ago

on the training side it is possible to convert the the js models to keras (h5) this means we can train models in the browser and load them into plugin

tensorflowjs_converter --input_format=tfjs_layers_model --output_format=keras models/ts9/model.json models/ts9/ts9.h5

taken from https://github.com/tensorflow/tfjs/tree/master/tfjs-converter#javascript-to-python

GuitarML commented 3 years ago

I think so, but for the plugin I was planning on using json format for the models anyway. H5 is just another data format, as long as all the weights are in the json file we can make it work. And we can work together to make sure the models trained from the browser can be loaded into the future plugin.

H5 is nice because it compresses the data, but I think json is the better option because it's readable in any text editor, and I'm more familiar with loading json in c++.

mishushakov commented 3 years ago

@GuitarML not sure if it's a good idea give SavedModel a try

once you have that, you can load the model in every version of tensorflow (you read it right, tensorflow, not just keras) in C++ specifically you can use the tensorflow C++ API or tensorflow Lite C++ API

if this is still isn't enough you could use tensorflow-onnx to convert your model to onnx and load it with onnx runtime

i've been actually wondering whether we can make both networks run in your plugin one solution would be adding wavenet to the tensorflow version the other solution would be converting both to onnx