Prompt Tuning met NaN error

bigscience-workshop / petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

https://petals.dev

MIT License

8.98k stars 498 forks source link

Open xinghua-qu opened 1 year ago

xinghua-qu commented 1 year ago

I tried to run the code for prompt tuning example, but got the NaN error after some iterations.

Does anybody know the reason for such error? Thanks :) The dots after the training curve mean NaN.

mryab commented 1 year ago

Hi @xinghua-qu! We made a fix for this in https://github.com/bigscience-workshop/petals/pull/343 which should likely resolve your issue. Can you try running the SST-2 prompt tuning notebook from main?