Reproduce track overfitting

[purzelrakete]

What

We would like to reproduce perfect track overfitting, as in #4. We expect the generated tracks to perfectly reproduce the training data.

Results

Write up the results of your experiment once it has completed and has been analysed. Include links to the treatment run, and also to the baseline if appropriate.

Acceptance Criteria

• [x] Run without batch norm
• [x] Run with batch norm

[purzelrakete]

Experiments

Baseline experiment notebook. Launched with:

bin/train track -p batch_norm False

We don't have a notebook for the baseline we are reproducing, so re-running without batch norm. This gets the same or slightly better results than the previous baseline for track. Rendered tracks sound like the input ✅

Treatment experiment notebook. Launched with:

bin/train track -p batch_norm True -p learning_rate 0.01

[purzelrakete]

Results

This turned into a bit of a batch norm experiment. That's because when we use the standard batch norm setup, the rendered audio no longer sounds like piano.

	train nll	test nll
baseline	0.0006	7.385
treatment	0.075	1.322

We can say that

• We managed to reproduce the baseline tracks result with perfect sounding audio generated
• Using batch norm, our loss is 100x higher on the trainset
• Using batch norm, our test loss is much lower

Insights

These results aren't very surprising. Batch norm has a regularising effect, which explains the better test likelihood, and also the worse training error.

Open questions

• How close are we to generating audio without overfitting?