search

richarddwang / electra_pytorch

Pretrain and finetune ELECTRA with fastai and huggingface. (Results of the paper replicated !)
324 stars 41 forks source link

Is it possible to perform the fine-tuning within the HuggingFace library? #28

Closed lucaguarro closed 3 years ago

lucaguarro commented 3 years ago

I have successfully performed the pretraining on my dataset using your Pretrain script and have my model saved to a checkpoint. Would it then be possible to load this model into the HuggingFace library so that I could use their Trainer class for performing the downstream task?

richarddwang commented 3 years ago

Since you used Huggingface model to pretrain, it should be possible. But I am not familiar with Trainer, maybe you can ask in their forum.

Feel free to tag me if you think there is anything else I could help.

lucaguarro commented 3 years ago

Well I suppose my concern comes from the fact that one should only load the pre-trained discriminator when performing fine-tuning on the classification task. My understanding is that the model loaded in my checkpoints folder is composed of both the generator and the discriminator. Is there a way where I can bypass the way the checkpoints are currently being saved and instead only save the discriminator by using huggingface save_pretrained function?

richarddwang commented 3 years ago

You can simply do electra_model.discriminator.save_pretrained(...) after learn.fit I think, and you can just ignore original checkpoints generated and take the huggingface checkpoint you saved after the training ends.