ICML 2020 reproducing issues

[yjlolo]

Thanks for your efforts on making the research reproducible. Colab is really a nice channel for the purpose. I have encountered some issues trying to reproduce the pitch detector, however, and would like to share with you.

Colab

• Modified the notebook to download a clip from MIR-1K and tested the mir1k checkpoint

• audio_ddsp_inv = synth(controls['harm_amp'], wavetable, controls['f0_hz']) incurs

  TypeError: Only integers, slices (`:`), ellipsis (`...`), tf.newaxis (`None`) and scalar tf.int32/tf.int64 tensors are valid indices, 
  got 'harm_amp'`

  indicating that controls is a tensor instead of a dictionary.

• Installing 0.14.0 fixes the problem. However, the checkpoint doesn't seem to work properly by comparing the extracted pitch contour with that by Crepe.

  

• More interestingly, re-running the Load a model cell with the same model specification from dropdown returns different predictions of the pitch contour, given the same tested audio.

I also tried to run the provided commands for evaluation in Colab. More specifically, the Colab downloads the provided URMP checkpoint and specify the provided gin files. This however leads to an error:

ValueError: No configurable matching reference '@data.Urmp()'. In binding for 'ZippedProvider.data_providers'.

The suspicion is that the checkpoints are not properly loaded, which could be related to #253. Also it would be great if you could provide the gin files for other datasets (there is only URMP at the moment if I understand correctly). The exact train/test splits and data pre-processing pipelines are not crystal clear in the paper, and it could be difficult to reproduce the results for other datasets.

[ben-hayes]

To add a little further colour to this issue — the same URMP error (ValueError: No configurable matching reference '@data.Urmp()'.) also occurs using the ddsp_run command with ddsp=0.13.0.