search

stanfordnlp / pyreft

ReFT: Representation Finetuning for Language Models
https://arxiv.org/abs/2404.03592
Apache License 2.0
947 stars 77 forks source link

[P1] catastrophic forgetting #78

Closed jiacheo closed 2 months ago

jiacheo commented 2 months ago

Good work. I saw your emoji demo and played with it for a time . I found it will never output English (or any other languages in text) whatever I let it to do. Is this the so called 'catastrophic forgetting' ? It seems that it has forgotten its mother language.

frankaging commented 2 months ago

@jiacheo thanks for your interests.

after finetuning (using ReFT or just vanilla SFT or LoRA or other PEFTs), the model becomes a specialized model as opposed to a general LM. It is adapted to certain domain so it forgets its pretraining ability.

if you want your model to only respond emojis to certain inputs, you have to come up with another training dataset! for example, only return emoji when there "dog" is mentioned.