search

meta-llama / codellama

Inference code for CodeLlama models
Other
15.92k stars 1.85k forks source link

Loss calculation always 0 #207

Open sanipanwala opened 7 months ago

sanipanwala commented 7 months ago

Hello,

I'm trying to fine-tune the 34B model but during fine-tuning, I always get a loss 0. While I was able to fine-tune 7B and 13B models but not 34B.

Let me know if I'm overlooking this or please give me suggestions.

Thanks.

jgehring commented 7 months ago

Hi @sanipanwala, we don't provide support for fine-tuning in this repository. Which tools are you using for this? Are you sure they support the 34B model well? The exact same setting works for 7B and 13B? In any case, a loss of 0 at the start of training is a good indication that something's going wrong.

sanipanwala commented 7 months ago

@jgehring I mean I'm using "codellama/CodeLlama-34b-hf" model and running a normal Python script and yes same configuration works with 7B and 13B.

Thanks.

sssszh commented 5 months ago

@sanipanwala Hi, have you solved this problem yet?

I found the same problem when trying to peft fine-tune CodeLLama-7B (using LlamaForSequenceClassification), the Loss is always 0 during the fine-tuning.

Thanks！

sanipanwala commented 5 months ago

Hi @sssszh ,

No, I haven't found any solution yet.

Thanks, Sani