Add ReZero, support pretraining from scratch

[Akamight]

I've trained two pretrained models for this new ReZero architecture, one with ~12000 piano midis, and another one fine tuned on an anime midi dataset. Both are trained on a 2080ti for around 7 days. Those pretrained models will be released soon on google drive once I find out a way to upload them.

ReZero really speeds up the training, and yields much better results (around 0.4 NLL for the pretrained model on its training set and ~0.2 NLL for the anime finetuned dataset). The expanded attention length while generating also yields more coherent long term structure.

RAdam is also used so an extra warmup operation doesn't need to be added. This might also be a factor in the better results achieved.

Note that this branch requires tqdm and joblib to run. Without these the preprocessing will be sluggishly slow.

Also note that the pretraining consumes a lot of system memory. Renting a VM on Vast.ai with their 512GB memory is recommended for pretraining.

For results achieved, here are some initial results. Some of the samples are conditioned with a primer, which is seperated from the generated samples by a very significant pause.

Heres the link: https://file.io/0Yo7dj

Further testing will be conducted, but this method seems extremely promising in music generation.

[remyhuang]

Thanks for your work! How about merging your code/results with a new readme file crediting to you into a new branch after you finnish the whole experiment and uploading? I will also mention your branch in the readme file of the master brach.

[Akamight]

That will be good to me! Thank you!

[remyhuang]

So, you seem to need to submit another PR with the complete content to the new branch (close this one) when you are ready? Let me know if you have other plans. Thanks!

[255doesnotexist]

R.I.P. for Akamight.