QLora + Mistral

[raghavlite]

I was trying qlora with mistral and I get this error.

    -m training.run \
    --output_dir /usr/project/xtmp/rt195/Sentence_Embedding/F5/gritlm/data/m7_temp \
    --model_name_or_path mistralai/Mistral-7B-v0.1 \
    --train_data /usr/project/xtmp/rt195/Sentence_Embedding/F5/gritlm/data/MEDI2/allnli.jsonl \
    --learning_rate 2e-5 \
    --lr_scheduler_type linear \
    --warmup_ratio 0.03 \
    --max_steps 1253 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 5 \
    --dataloader_drop_last \
    --normalized \
    --temperature 0.02 \
    --train_group_size 2 \
    --negatives_cross_device \
    --query_max_len 256 \
    --passage_max_len 2048 \
    --mode embedding \
    --logging_steps 1 \
    --bf16 \
    --pooling_method mean \
    --attn cccc \
    --attn_implementation sdpa \
    --save_steps 5000 \
    --gradient_checkpointing \
    --qlora

if i change the command to --lora instead of --qlora, it works.

Any help is appreciated. Also, what version of peft and bitsandbytes are you using?

[Muennighoff]

Hm I havn't tried the lora & qlora integrations

[raghavlite]

thanks