[FEATURE]: Llama2 70B finetune example

[tangzhy]

Describe the feature

Dear,

I assume many developers have the same request as I had, that is to finetune Llama2 70b beyond a single node's capability.

I wonder do you have a full example of fine-tuning Llama2 70b and then saving to hf compatible model.

Currently, you only have a benchmark.py script that may not target fine-tuning demand and I see no saving in that example.

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: [FEATURE]: Llama2 70B finetune example

[ver217]

Hi, we've provided a pretraining (from scratch) script pretrain.py. Finetuing script will be added later.

[jacklanda]

Comment for Better Reach

[adivoj]

Also waiting for this. Would appreciate the example! Greatvwork BTW.

[LXXiaogege]

The llama2 fine-tuning example doesn't have the option to add peft, it looks like a full parameter fine-tuning, will this be added later?

[flybird11111]

Thank you for your suggestion, we will consider adding peft in the future.

[LXXiaogege]

Thank you for your suggestion, we will consider adding peft in the future.

if I want to implement lora de-tuning in finetune.py, do I just need to add the following code:

    from peft import LoraConfig, TaskType,get_peft_model
    peft_config = LoraConfig(task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32,
                             lora_dropout=0.1)
    model = get_peft_model(model, peft_config)
    model.print_trainable_parameters()

[LXXiaogege]

When I run finetune.py for LLama2 fine tuning, the loss stays as Nan, where is this problem coming from? Is there any good solution for this?