binmakeswell / ColossalChat

ColossalChat is the project to implement LLM with RLHF, powered by the Colossal-AI project.
https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat
Apache License 2.0
61 stars 17 forks source link

Is there a typo here? #5

Closed digger-yu closed 1 year ago

digger-yu commented 1 year ago

with

\ColossalChat\examples\train_sft.sh \ColossalChat\README.md line 125 etc.

torchrun --standalone --nproc_per_node=4 train_sft.py \
    --pretrain "/path/to/LLaMa-7B/" \
    --model 'llama' \
    --strategy colossalai_zero2 \
    --log_interval 10 \
    --save_path  /path/to/Coati-7B \
    --dataset /path/to/data.json \
    --batch_size 4 \
    --accimulation_steps 8 \
    --lr 2e-5 \
    --max_datasets_size 512 \
    --max_epochs 1 \

Are the accimulation_steps parameters here correct?