[ddsp] Use our own trained model

[pranayaggarwal]

Hi Team,

We have trained our own ddsp timbre transfer model using these steps

✅ This model is working fine with the notebook version of custom model load i.e. https://colab.research.google.com/github/magenta/ddsp/blob/master/ddsp/colab/demos/timbre_transfer.ipynb

However, we were trying to plug this in magenta-js

• Tried playing in - https://github.com/magenta/magenta-js/tree/master/music/demos ❌ It doesn't work. The output audio is simply nothing, it doesn't have sound.

We have used following

• For exporting model to TFJS - https://github.com/magenta/ddsp/blob/main/ddsp/training/ddsp_export.py

Could you please help us with this issue and help with fixing perhaps any broken steps, if with current versions?

This is our model.json file https://gist.github.com/pranayaggarwal/fc4503673989ffd48c03e2dec9c9d6e4#file-model-json

Note:

Few things we have noticed while debugging on our own are that -

1. Existing ddsp models used on https://magenta.github.io/magenta-js/music/demos/ddsp_tone_transfer.html have different structure of model.json where DT_FLOAT of single "size": "80000 is used where our model.json created by ddsp_export.py doesn't have this identical structure and it uses DT_FLOAT of various sizes.

2. Voilin's model.json https://storage.googleapis.com/magentadata/js/checkpoints/ddsp/violin/model.json

   {"format": "graph-model", "generatedBy": "2.4.0", "convertedBy": "TensorFlow.js Converter v1.7.0"

Our model.json by generated by ddps_export has

{"format": "graph-model", "generatedBy": "2.5.0", "convertedBy": "TensorFlow.js Converter v3.8.0"

[pranayaggarwal]

cc @jesseengel , @notwaldorf

[pranayaggarwal]

cc @edwintoh

[jesseengel]

Sorry for the delay on this. This code path wasn't very robust and probably was broken somewhere along the way. There is some flux with the ddsp repo at the moment (fixing a bug with the power/loudness calculation), so part of fixing that will be making sure this export works. It might take a little bit (hopefully not more than a couple weeks max [sorry, lots of competing priorities]) but it's actively on the TODO list.