Training using pre-trained performanceRNN weights

[karolmajek]

How to load pre-trained performanceRNN model weights (.mag file with or without dynamics) to start learning from these weights? I want to retrain performanceRNN on my small midi set (only 60 midi files, too little to learn from scratch)

[iansimon]

You should be able to use this script to extract a checkpoint from one of our bundle files: https://github.com/tensorflow/magenta/blob/master/magenta/scripts/unpack_bundle.py

And then run performance_rnn_train pointing --run_dir at wherever the extracted checkpoint lives.

[zZyan]

@iansimon Hi Ian, I am encountering the same issue and tried the method you suggested. However, it seems that the model is still training from the begining, i.e. it doesn't mention about restoring saved checkpoint as I will expect. Also, I noticed that runing performance_rnn_train, .index and .data files will be saved besides .meta and .ckpt files. I am not sure whether I will need those other two files while restoring checkpoint. Thank you!

Below is the output when runing performance_rnn_train as you suggested,
INFO:tensorflow:Train dir: performance_rnn/run2\train INFO:tensorflow:Starting training loop... INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Saving checkpoints for 1 into performance_rnn/run2\train\model.ckpt.

while I am expecting something like this: INFO:tensorflow:Restoring parameters from performance_rnn/run3\train\model.ckpt

[iansimon]

Can you send me the exact command you entered?

[zZyan]

Update on the issue.

SEQUENCES_TFRECORD=data/improv_rnn/jazz_xml2/training_lead_sheets.tfrecord RUNDIR=improv_rnn/run6

improv_rnn_train \ --config=chord_pitches_improv \ --run_dir=${RUNDIR} \ --sequence_example_file=${SEQUENCES_TFRECORD} \ --hparams="batch_size=128,rnn_layer_sizes=[256, 256, 256]" \ --num_training_steps=10

command: ls improv_rnn/run6/train output: checkpoint events.out.tfevents.1511859596.DESKTOP-VTMLC08 graph.pbtxt model.ckpt model.ckpt.meta model.ckpt-1.data-00000-of-00001 model.ckpt-1.index model.ckpt-1.meta

Many thanks. Yan

[iansimon]

One step towards a fix is that after unpacking the bundle but before training you should create a file called "checkpoint" in improv_rnn/run6/train that looks like:

model_checkpoint_path: "model.ckpt"
all_model_checkpoint_paths: "model.ckpt"

But there seems to be some kind of checkpoint incompatibility with what's in the bundle; in fact even if you create a new bundle then unpack it and train you'll get variable name errors. I'm looking into it now...

[zZyan]

That's right. I tried adding checkpoint file but it doesn't work. Thank you for your help!

Regards, Yan

[ianni67]

Hello, sorry for bumping up this post. I also want to experiment with refining a pre-trained net and, thus, would like to know it the issue with checkpoints compatibility has been solved.

[Ignasinou]

Hi, sorry for bumping up this post too. I am working with performance_rnn pre-trained models and I am in a similar stage that @zZyan was. I used the unpack_bundle function and then training the model but the parameters are not restored from the downloaded checkpoint. I tried the indications previously posted. Is there any workaround or any news about it @iansimon? Regards.

[iansimon]

Amazingly, it looks like fine-tuning has never worked for our RNN models, or has at least been broken for years. #1753 (in progress but the code should work right now) fixes this and supports fine-tuning from a bundle file for Performance RNN.

[Ignasinou]

@iansimon Thank you very much, I just performed the training from a bundle file for Performance RNN and it seems that everything works as expected.