Closed lloo099 closed 1 year ago
I think you can replace the --wandb_entity
to your personal information. You can refer the wandb documentation for details. We use ntu-slab
since it's our team project. For personal use, you can refer to ur own identity.
I think you can replace the
--wandb_entity
to your personal information. You can refer the wandb documentation for details. We usentu-slab
since it's our team project. For personal use, you can refer to ur own identity.
Thank you. The solution you provided worked. However, I am experiencing issues with the CUDA memory. I am currently running the Otter-LLaMa7B model on three 3090 GPUs, but it is running out of memory. Do you happen to know how I can make it run on three 3090 GPUs? I have already reduced the batch size and am using bf16.
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml \
pipeline/train/instruction_following.py \
--pretrained_model_name_or_path=/home/jjc/Otter/OTTER-LLaMA7B-Init \
--mimicit_path="/home/jjc/Otter/mimic-it/mimicit_data/LA/LACONV_instructions.json" \
--images_path="/home/jjc/Otter/mimic-it/convert-it/output/LA.json" \
--train_config_path="/home/jjc/Otter/mimic-it/mimicit_data/LA/LACONV_train.json" \
--batch_size=1 \
--num_epochs=1 \
--report_to_wandb \
--wandb_entity=hkujjc \
--run_name=OTTER-LLaMA7B-densecaption \
--wandb_project=OTTER-LLaMA7B \
--workers=1 \
--lr_scheduler=cosine \
--learning_rate=1e-5 \
--warmup_steps_ratio=0.01 \
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 23.70 GiB total capacity; 22.31 GiB already allocated; 5.75 MiB free; 22.39 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Can you try to make sure you modified the OtterForConditionalGeneration
related code in instruction_following.py
?
You need to pass a **precision
args to make sure it's loaded in bf16
version.
You could project-wise search **precision
to see how we handle it.
Can you try to make sure you modified the
OtterForConditionalGeneration
related code ininstruction_following.py
?You need to pass a
**precision
args to make sure it's loaded in thebf16
version.You could project-wise search
**precision
to see how we handle it.
Cool, thanks. I can load the 16-bit model following ur **precision settings. But the input dtype doesn't match with 16b. As I already changed the dtype of autocast() and cast().
You could also follow our code in demo, to set the dtype
of input tensors.
Thanks
Hi, I encounter a compile problem with wandb. My wandb account was signed and released an open project here (https://wandb.ai/zhoutomas177/OTTER-LLaMA7B). But the error was shown when I run the code:
API Error is here:
Does anyone encounter this issue glad to help me. Thanks,