search

lucidrains / reformer-pytorch

Reformer, the efficient Transformer, in Pytorch
MIT License
2.13k stars 256 forks source link

Music Reformer results (some problems) #135

Open asigalov61 opened 3 years ago

asigalov61 commented 3 years ago

@lucidrains Tried reformer as you have suggested...Results were also not very impressive, unfortunately... In fact, I am getting much better results with the official Google implementation of Reformer so maybe the performance problems are due to something missing in the implementation? Again, not sure...

Here are the results:

NUM_BATCHES = int(1e5) BATCH_SIZE = 20 GRADIENT_ACCUMULATE_EVERY = 4 LEARNING_RATE = 3e-4 VALIDATE_EVERY = 100 GENERATE_EVERY = 500 GENERATE_LENGTH = 1024 SEQ_LEN = 2048

training: 1%| | 1173/100000 [3:01:10<251:14:04, 9.15s/it]training loss: 1.5287952423095703

Generation 2048 @ 0.8

See attached samples.

It lost coherence mid-way on me in each one :( Not sure why it is so bad :( Original Google implementation works fine.

Music_Reformer_MIDI_Output_Samples.zip

KMzuka commented 2 years ago

I found a similar problem when training with REMI, the prediction of MIDI events is not very accurate...