Is it possible finetune llama3 with torchtune

[arifsaeed]

Hi,

Im struggling a little with getting huggingface trainer working for my use case. is it possible to fine tune a llama3 model with a trainer i write with torch tune. Looking at huggingface it looks like it uses pytorch under the hood anyway. Any direction on how to achieve this would be greatly appreciated.

if i setup my model as like below, could this be run via a torch training script model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/llama-3-8b-Instruct", max_seq_length=max_seq_length, dtype=dtype, load_in_4bit=load_in_4bit, token=get_hf_token(hf_licencepath), )

model = FastLanguageModel.get_peft_model(
    model,
    r=16,  # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,  # Supports any, but = 0 is optimized
    bias="none",  # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long context
    random_state=3407,
    use_rslora=False,  # We support rank stabilized LoRA
    loftq_config=None,  # And LoftQ
)

[danielhanchen]

Sorry currently we specifically wrap the Hugging Face trainer