Open Edenzzzz opened 5 months ago
Hi, I took the hyperparams from the paper but got only 32.1 MMLU acc. Could you point out what could be wrong here? I've also attached training logs. Thanks!
python qlora.py \ --model_name_or_path huggyllama/llama-7b \ --use_auth \ --output_dir /fly/results/qlora \ --logging_steps 10 \ --save_strategy steps \ --data_seed 42 \ --save_steps 500 \ --save_total_limit 40 \ --evaluation_strategy steps \ --eval_dataset_size 1024 \ --max_eval_samples 1000 \ --per_device_eval_batch_size 1 \ --max_new_tokens 32 \ --dataloader_num_workers 1 \ --group_by_length \ --logging_strategy steps \ --remove_unused_columns False \ --do_train \ --do_eval \ --do_mmlu_eval \ --lora_r 64 \ --lora_alpha 16 \ --lora_modules all \ --double_quant \ --quant_type nf4 \ --bf16 \ --bits 16 \ --warmup_ratio 0.03 \ --lr_scheduler_type constant \ --gradient_checkpointing \ --dataset alpaca \ --source_max_len 16 \ --target_max_len 512 \ --per_device_train_batch_size 1 \ --gradient_accumulation_steps 16 \ --max_steps 1875 \ --eval_steps 187 \ --learning_rate 0.0002 \ --adam_beta2 0.999 \ --max_grad_norm 0.3 \ --lora_dropout 0.1 \ --weight_decay 0.0 \ --seed 0 \ --mmlu_split test
@artidoro
Hi, I took the hyperparams from the paper but got only 32.1 MMLU acc. Could you point out what could be wrong here? I've also attached training logs. Thanks!
@artidoro