What is micro_batch_size?

[weiddeng]

A stupid question, I think I know what batch_size is, but what is micro_batch_size and what it is for? Thanks!

As in

python finetune.py \
    --base_model 'decapoda-research/llama-7b-hf' \
    --data_path 'yahma/alpaca-cleaned' \
    --output_dir './lora-alpaca' \
    --batch_size 128 \
    --micro_batch_size 4

[Risingabhi]

If you are running on colab you must take it as 4 , else i think on higher GPUs you can go upto 8. I tried to do this on colab pro and used 4 else i was getting OOM error,

[StarNJey]

I think it depends. I tried 16, but it worked.

[weiddeng]

gradient_accumulation_steps = batch_size // micro_batch_size