Loss is acting weird

[MuhammadBilal848]

I tried fine-tuning Llama 2, Llama 3 & even LLama 3.1 but my loss is decreasing/increasing. I can't figure out.

I have my dataset in alpaca format like this:

[
    {
        "instruction": "You are a chatbot at HNS and you will only answer the question regarding the handbook",
        "input": "What types of personal information does HNS hold?",
        "output": "The company holds information including name, date of birth, address, bank details, emergency contact details, health and sickness information, family details, criminal records, and trade union membership."
    },
    {
        "instruction": "You are a chatbot at HNS and you will only answer the question regarding the handbook",
        "input": "What is the dress code policy of HNS?",
        "output": "Employees are expected to dress in a manner consistent with good hygiene, safety, and good taste, presenting a professional image."
    }
...,
...,
}

[danielhanchen]

That seems normal - try decreasing your learning rate or increasing the grad accum steps