Errors with loading custom trained melody_rnn models into browser

[mjxmusic]

Hi this is MJ - I've been attempting to load my custom trained model (that I know outputs melodies and training works based on my manual method) to respond to the MagentaJS tutorial. When I run through MagentaJS tutorial (e.g. https://hello-magenta.glitch.me/) I am able to get your model to respond via browser (multiple of your models I've tried w/ success).

The issue is when it is my model, I've trained the model using the following configuration:

melody_rnn_train \ --config=attention_rnn \ --run_dir=/ml-music/run_dir_melody \ --sequence_example_file=~/ml-music/data_for_training/all_melody_notesequences.tfrecord \ --hparams="batch_size=64,rnn_layer_sizes=[64,64]" \ --num_training_steps=20000

and I get a successful after training on my computer or VM (which is where training took place) and am able to output desired MID files by using:

melody_rnn_generate \ --config=attention_rnn \ --run_dir=/ml-music/run_dir_melody \ --output_dir=/ml-music/outputs \ --num_outputs=10 \ --num_steps=500 \ --hparams="batch_size=64,rnn_layer_sizes=[64,64]" \ --primer_melody="[60]"

The issue I have begins when I try implementing this model within Magenta JS on my HTML webpage - when looking at the console, the issue I get is the following:

^IMG above, but incase there are issues seeing, the error I get within my Chrome console after running is "Uncaught (in promise) TypeError: Cannot read property 'matMul' of undefined at t.sampleRnn (music@^1.0.0:83) at music@^1.0.0:83 at music@^1.0.0:22 at t.scopedRun (music@^1.0.0:22) at t.tidy (music@^1.0.0:22) at Module.$e (music@^1.0.0:22) at t. (music@^1.0.0:83) at music@^1.0.0:83 at Object.next (music@^1.0.0:83) at music@^1.0.0:83"

Another thing worth noting (not sure if this affects anything), but when running the conversion step (https://github.com/magenta/magenta-js/blob/master/scripts/checkpoint_converter.py) on my checkpoint, the folder that I end up pointing to for the Magenta model contains this type of data --

Only adding if this helps w/ debugging - content of weights_manifest.json below:

[{"weights": [{"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_1/basic_lstm_cell/bias", "shape": [256]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/attention/attn_v", "shape": [64]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/attention/bias", "shape": [64]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/attn_output_projection/bias", "shape": [64]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/basic_lstm_cell/bias", "shape": [256]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_1/basic_lstm_cell/kernel", "shape": [128, 256]}, {"dtype": "float32", "name": "fully_connected/weights", "shape": [64, 40]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/kernel", "shape": [138, 74]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/attention/attn_w", "shape": [1, 1, 64, 64]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/attn_output_projection/kernel", "shape": [128, 64]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/basic_lstm_cell/kernel", "shape": [138, 256]}, {"dtype": "float32", "name": "fully_connected/biases", "shape": [40]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/attention/kernel", "shape": [128, 64]}, {"dtype": "float32", "name": "rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/bias", "shape": [74]}], "paths": ["group1-shard1of1.bin"]}]

[notwaldorf]

I think the problem might be that the JS music RNN doesn't implement attention (@adarob do you know?)

What happens if you use --config=basic_rnn instead?

[notwaldorf]

Oh hmm, the JS MusicRNN seems to have some code about attention. I'm going to defer to Adam on this one since he knows the model better

[adarob]

I think there is an unfortunate incompatibility that was introduced when we changed how the attention was applied in MusicRNN (https://github.com/tensorflow/magenta/pull/670/files). We need to update the JS model (I'm not sure how difficult it would be), but for now you could retrain your model the "old" way.

[ganeshkumartk]

Hey, Can I adopt this issue for GSoC 2020 ? If so, I'm happy to contribute.

[adarob]

It would be great to have this help! I'm not sure how to set it up as a GSoC project, but please feel free to pursue it!

On Thu, Mar 26, 2020 at 7:43 AM Ganesh kumar T K notifications@github.com wrote:

Hey, Can I adopt this issue for GSoC 2020 ? If so, I'm happy to contribute.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/magenta/magenta-js/issues/427#issuecomment-604470753, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIJV2BQGIIGVV45WEL62M3RJNS2DANCNFSM4LEHXMOQ .

[ganeshkumartk]

Yeah, I'm working on reconstructing the layers

[mjxmusic]

Hey folks, wanted to just bump this to see if there were any updates?

[mjxmusic]

Hi @adarob - I wanted to see if there were any updates on this? And in case I wanted to pursuit training the model "the old way", could you elaborate what you mean by that?

[adarob]

You would need to switch where the attention is applied (basically revert that PR mentioned above).