Open zuwenqiang opened 1 year ago
And during training, the system prompts me with "Using dtype torch.float32."
I think the GPU usage looks normal but you could try with deepspeed_zero2 config and set the num_processes=2.
using zero2 will be much faster than fsdp if you have 80G A100.
Hi! I'm using two A100 GPUs, each with 40GB of memory. This is the GPU memory utilization for my training. I'm almost reaching over 90% memory utilization on both A100 GPUs.![image](https://github.com/Luodian/Otter/assets/55320101/86a8fa50-6a73-4cab-84bd-3d8bf83b054c)
Is this normal? Here's the configuration: accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml \ pipeline/train/instruction_following.py \ --pretrained_model_name_or_path=luodian/OTTER-LLaMA7B-INIT \ --mimicit_path="path/to/SN_instruction.json" \ --images_path="path/to/SN.json" \ --batch_size=4 \ --num_epochs=9 \ --report_to_wandb \ --wandb_entity=ntu-slab \ --run_name=OTTER-LLaMA7B-densecaption \ --wandb_project=OTTER-LLaMA7B \ --workers=1 \ --lr_scheduler=cosine \ --learning_rate=1e-5 \ --warmup_steps_ratio=0.01
and here's the accelerate config(accelerate_config_fsdp.yaml): compute_environment: LOCAL_MACHINE distributed_type: no downcast_bf16: false machine_rank: 0 main_training_function: main mixed_precision: fp16 num_machines: 1 num_processes: 1 rdzv_backend: static same_network: false tpu_use_cluster: false tpu_use_sudo: false use_cpu: false main_process_port: 20687