search

kanekomasahiro / context-debias

MIT License
25 stars 3 forks source link

debias gpt2 model #3

Open Ewanwong opened 1 year ago

Ewanwong commented 1 year ago

Hi, I'm trying to apply context-debias to generative models (gpt2). I tried to directly use your script but the loss is extremely large (1e10+). I notice you have an argument "mlm" in run_debias_mlm.py, but it doesn't seem to be used in the code, so I wonder whether there is something in the code I need to modify to adjust for causal language models? Thank you!

Ewanwong commented 1 year ago

and I notice the large losses are from neutral dataset, do you think this is the property of gpt2 embeddings or it can be fixed by adjusting codes?