search

facebookresearch / music-translation

A UNIVERSAL MUSIC TRANSLATION NETWORK - a method for translating music across musical instruments and styles.
Other
459 stars 71 forks source link

Saving checkpoints for single node training #15

Open bdwivedi6 opened 4 years ago

bdwivedi6 commented 4 years ago

From my understanding, the number of models saved in the checkpoints folder should be equal to the number of datasets. In the case of single node training, self.args.rank always remains 0, in which case all the checkpoints get named "bestmodel_{self.args.rank}.pth ", which would always be "bestmodel_0.pth". Am I missing something here? Thank you so much!

18praveenb commented 4 years ago

See #7, this is a bug. A modified version of train.py that might fix it is at https://github.com/18praveenb/orchestrator/blob/master/music-translation/src/train.py however I haven't finished testing it yet.