search

ohmeow / blurr

A library that integrates huggingface transformers with the world of fastai, giving fastai devs everything they need to train, evaluate, and deploy transformer specific models.
https://ohmeow.github.io/blurr
Apache License 2.0
289 stars 34 forks source link

Best way to use a finetuned/customised LM for downstream task e.g. sequence classification? #48

Closed yijinlee closed 3 years ago

yijinlee commented 3 years ago

Hi,

Starting from pretrained GPT2 hf_model, I finetuned a custom LM using unlabelled dataset, using the model_cls = AutoModelForCausalLM method, and have saved/exported the resulting Learner object (let's call that finetuned_LM_learn).

What is the best way for me to then load that finetuned LM, and use it for downstream task (e.g. sequence classification) on labelled dataset? Should I just go through the same steps here, and then switch out the base_model before starting to train? Something like below?

learn.model.hf_model.base_model = finetuned_LM_learn.model.hf_model.base_model
learn.fit_one_cycle(epoch, lr)

Or is that not the correct / best way? Thanks.

yijinlee commented 3 years ago

Ok I think I should be using finetuned_LM_learn.hf_model.save_pretrained(path) and then for downstream classification task do AutoModelForSequenceClassification.from_pretrained(path) instead. Please do correct me if I am wrong.. Thank you.

ohmeow commented 3 years ago

Yup. That is correct.

Just use the transformer model's save_pretrained to the path you want to save your HF artifacts in (you should do the same with the tokenizer) and the use as you are above.