theobjectivedad
commented
1 year ago Hi @utensil, I'm working with DeepSpeed and am having similar issues. Although my process is still broken, I'll share my current config in case it helps. For testing, I have been able to start the the training process on 1 node w/ 3x A6000s under zero 2. Here is my Makefile target:
WORKSPACE_HOST_PATH:=...
MODELS_HOST_PATH:=...
DATA_HOST_PATH:=...
WORK_HOST_PATH:=...
train:
docker run --gpus='all' -it --rm \
--volume=$(WORKSPACE_HOST_PATH):/workspace \
--volume=$(MODELS_HOST_PATH):/models \
--volume=$(DATA_HOST_PATH):/data \
--volume=$(WORK_HOST_PATH):/work \
--volume=$(WORKSPACE_HOST_PATH)/extern/axolotl:/opt/axolotl \
--env-file=$(CURDIR)/.env \
--entrypoint=accelerate \
quay.io/theobjectivedad/axolotl-main:latest \
launch \
--config_file /work/accelerate/basic.yaml \
/opt/axolotl/scripts/finetune.py \
/work/atheos/config.yamlMy accelerate config:
compute_environment: LOCAL_MACHINE
deepspeed_config:
deepspeed_config_file: /SET_IN_AXOLOTL_CONFIG.yaml
zero3_init_flag: false
distributed_type: DEEPSPEED
downcast_bf16: "no"
machine_rank: 0
main_training_function: main
num_machines: 1
num_processes: 3
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: falseDeepspeed config:
{
"optimizer": {
"type": "auto"
},
"scheduler": {
"type": "auto"
},
"activation_checkpointing": {
"partition_activations": "auto"
},
"zero_optimization": {
"stage": 2,
"offload_optimizer": {
"device": "auto"
},
"offload_param": {
"device": "auto"
},
"allgather_bucket_size": "auto",
"allgather_bucket_dtype": "auto",
"dp_bucket_size": "auto",
"overlap_comm": "auto",
"contiguous_gradients": "auto",
"sub_group_size": "auto",
"reduce_bucket_size": "auto"
},
"gradient_clipping": "auto",
"fp16": {
"enabled": "auto"
},
"bf16": {
"enabled": "auto"
},
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto"
}Axolotl config:
###############################################################################
# Model
###############################################################################
base_model: /models/llama-7b-hf
base_model_config: /models/llama-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
output_dir: /work/atheos/output1
sequence_len: 2048
max_packed_sequence_len: 1024
tokens:
bos_token: "<s>"
eos_token: "</s>"
unk_token: "<unk>"
pad_token: "<unk>"
special_tokens:
###############################################################################
# Precision & Model loading
###############################################################################
bf16: full
bfloat16: true
fp16: false
float16: false
tf32: true
load_in_8bit: false
load_in_4bit: false
lora_model_dir:
###############################################################################
# Dataset
###############################################################################
datasets:
- path: /data/GPTeacher/Instruct
type: gpteacher
dataset_prepared_path: /work/last_run_prepared
val_set_size: 0.02
###############################################################################
# Training
###############################################################################
deepspeed: /work/accelerate/ds_stage2_auto.json
adapter: lora
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
- q_proj
- v_proj
lora_fan_in_fan_out: false
# WanDB configuration
wandb_project: smoketest
wandb_watch:
wandb_run_id:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 8
optimizer:
torchdistx_path:
lr_scheduler: cosine
learning_rate: 5.0e-5
train_on_inputs: false
group_by_length: false
early_stopping_patience: 3
auto_resume_from_checkpoints: true
resume_from_checkpoint:
logging_steps: 500
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 20
eval_steps: 500
save_steps: 500
debug: false
weight_decay: 0.1
fsdp:
fsdp_config:
utensil
casper-hansen
Aillian
grimulkan
noobmaster29
I've been trying to make the combination
deepspeed + qlora + falconwork but due to unknown reasons I've stuck in an error maze.Setup
winglian/axolotl-runpod:main-py3.9-cu118-2.0.0bash -c "curl -H 'Cache-Control: no-cache' https://raw.githubusercontent.com/utensil/llm-playground/main/scripts/entry/prepare_ax.sh -sSf | bash"ds_config.json(final version, modified from the default one in axolotl):examples/falcon/config-40b-qlora.ymlpush_dataset_to_hub: utensil hf_use_auth_token: true
datasets:
dataset_prepared_path: last_run_prepared val_set_size: 0.01
enable QLoRA
adapter: qlora lora_model_dir: sequence_len: 2048 max_packed_sequence_len:
hyperparameters from QLoRA paper Appendix B.2
"We find hyperparameters to be largely robust across datasets"
lora_r: 64 lora_alpha: 16
0.1 for models up to 13B
0.05 for 33B and 65B models
lora_dropout: 0.05
add LoRA modules on all linear layers of the base model
lora_target_modules: lora_target_linear: true lora_fan_in_fan_out:
wandb_project: falcon-qlora wandb_watch: wandb_run_id: wandb_log_model: output_dir: /content/axolotl-trained/falcon-qlora-40b-gsm8k/
QLoRA paper Table 9
- 16 for 7b & 13b
- 32 for 33b, 64 for 64b
Max size tested on A6000
- 7b: 40
- 40b: 4
decrease if OOM, increase for max VRAM utilization
micro_batch_size: 1 gradient_accumulation_steps: 1 num_epochs: 3
Optimizer for QLoRA
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
QLoRA paper Table 9
- 2e-4 for 7b & 13b
- 1e-4 for 33b & 64b
learning_rate: 0.0002 train_on_inputs: false group_by_length: false bf16: true fp16: false tf32: true gradient_checkpointing: true
stop training after this many evaluation losses have increased in a row
https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3 resume_from_checkpoint: auto_resume_from_checkpoints: true local_rank: logging_steps: 1 xformers_attention: true flash_attention: gptq_groupsize: gptq_model_v1: warmup_steps: 10 eval_steps: 5 save_steps: 10 debug: deepspeed: weight_decay: 0.01 fsdp: fsdp_config: special_tokens: pad_token: "<|endoftext|>" bos_token: ">>ABSTRACT<<" eos_token: "<|endoftext|>"
Setting ds_accelerator to cuda (auto detect)
DeepSpeed C++/CUDA extension op report
NOTE: Ops not installed will be just-in-time (JIT) compiled at runtime if needed. Op compatibility means that your system meet the required dependencies to JIT install the op.
JIT compiled ops requires ninja ninja .................. [OKAY]
op name ................ installed .. compatible
async_io ............... [YES] ...... [OKAY] cpu_adagrad ............ [YES] ...... [OKAY] cpu_adam ............... [YES] ...... [OKAY] fused_adam ............. [YES] ...... [OKAY] fused_lamb ............. [YES] ...... [OKAY] quantizer .............. [YES] ...... [OKAY] random_ltd ............. [YES] ...... [OKAY] [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0 [WARNING] using untested triton version (2.0.0), only 1.0.0 is known to be compatible sparse_attn ............ [NO] ....... [NO] spatial_inference ...... [YES] ...... [OKAY] transformer ............ [YES] ...... [OKAY] stochastic_transformer . [YES] ...... [OKAY] transformer_inference .. [YES] ...... [OKAY] utils .................. [YES] ...... [OKAY]
DeepSpeed general environment info: torch install path ............... ['/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch'] torch version .................... 2.0.1+cu118 deepspeed install path ........... ['/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/deepspeed'] deepspeed info ................... 0.9.3+52907a66, 52907a66, master torch cuda version ............... 11.8 torch hip version ................ None nvcc version ..................... 11.8 deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8