[MusicBERT] Compatibility with latest fairseq repo

microsoft / muzic

Muzic: Music Understanding and Generation with Artificial Intelligence

MIT License

4.46k stars 434 forks source link

[MusicBERT] Compatibility with latest fairseq repo #61

Open tripathiarpan20 opened 2 years ago

tripathiarpan20 commented 2 years ago

Hi!

I have been thinking about experimenting with latest extensions/features of PyTorch and for that the MusicBERT code would need to be compatible with the latest version of fairseq repo (which is presumably compatible with latest PyTorch repo too).

Is it possible to make the changes in the code soon for compatibility with latest fairseq repo? (v 0.12.2) I would have tried to do that myself and raised a PR but currently I got my hands full.

Thanks!

mlzeng commented 2 years ago

Hi @tripathiarpan20

Migrating the code should be easy. But migrating the pre-trained checkpoints is tricky.

The MusicBERT plugin overrides a few methods of the original RoBERTa-related classes. If the original interfaces of methods are changed, then you may need to modify the code to adapt to the new interfaces.

tripathiarpan20 commented 2 years ago

I see, I shall try doing these actually and update in a few days!.

tripathiarpan20 commented 2 years ago

Basically what I have noticed (for the RobertaEncoder class) is that:

In the fairseq v0.10.1 implementation, TransformerSentenceEncoder (inherited by OctupleEncoder) encapsulated both the sentence embeddings as well as the Transformer encoder layers.
Meanwhile, the v0.12.2 implementation encapsulates sentence embedding into the default PyTorch torch.nn.Embedding class and the Transformer encoder layers into TransformerEncoder class.

Porting the checkpoints should also be possible after making changes to the current MusicBERT plugin.