How can we pre-train from scratch?

[neel04]

Is there any way I can use this repo to pre-trained Fnet from scratch?

[erksch]

Yes! We have code for PyTorch pre-training for the FNet model, if you want we can also open-source it.

[neel04]

Thanks for the quick reply - and yes, that would be lovely! :hugs:

[erksch]

Okay, you may have to wait a little bit because the training code is in another project and we need to check out how to provide it here (either separate train repo or code here).

[erksch]

Btw you can always use the training code from the official implementation and just convert the checkpoint. If you are fine with the type of tokenizer they are using.

[neel04]

yeah :sweat_smile: the main problem was the tokenizer they were using - and they haven't given any instructions on how I can use my own custom one :(

[erksch]

Shouldn't be too difficult, just search for tokenizer in the code and replace all calls with the interface of your tokenizer e.g. a huggingface tokenizer. Most tokenizers have pretty similar APIs

[erksch]

But in our training code, you can also use HuggingFace tokenizers or just a custom WordPiece vocab

[neel04]

I would prefer if you have training scripts (since there is already a module there) because it doesn't seem the quickest idea to modify such a large repo of Fnet...

[neel04]

I am trying to get it to work with lucidrains' MLM pytorch repo, but It doesn't seem to work :(

[erksch]

Give me an hour or two and I'll have a branch with the training code. I'd be glad if you tried it out

[neel04]

Great! Appreciate you putting in the effort for some stranger online :+1: :cake: :partying_face:

[erksch]

You can check out the pre-training code on https://github.com/erksch/fnet-pytorch/tree/pretraining-code/training

[neel04]

Thanks a lot @erksch for that, I think I can muddle along from here! :hugs:

[erksch]

Alright! If you like you could give the repo a star :P

[neel04]

sure! :grin: