-
Cant understand, how is num_epochs calculated.
I use, for example, 16 images, 1 repeat, 5 epochs, 1 batch size, 16 grad accum steps. And i get 40 epoch.
Why batches per epoch is 32?
What? 😢
Help …
-
Hello,
I'm glad to hear that you've achieved good results and are planning further research based on this work. However, you're currently facing challenges with the pre-training process using the c…
-
Hi,
We'are trying to use DPO to align our fine-tuned Falcon-7B model.
We actually already have fine-tuned falcon-7b model, so I used SFT on a small sample ~1000 observations, and generated policy…
-
Hello,
Thank you for sharing your work!
I noticed that the `finetune.sh` and `finetune_fsdp.sh` use regular training by default. Should I change it to `zo` to enable MeZO trainer? Also, I'm gett…
-
I am trying to run the scripts you provide in "Huggingface Accelerate Integration of Deepspeed"
`accelerate launch --config_file configs/accelerate/dist_8gpus_zero3_offload.yaml main_accelerate.py -…
-
配置:
!torchrun --nproc_per_node 1 \
-m FlagEmbedding.reranker.run \
--output_dir /bge-reranker-v2-m3-finetune \
--model_name_or_path /bge-reranker-v2-m3/bge-reranker-v2-m3 \
--train_data output.js…
-
Tried to load and distribute model to devices in a layerwise way, by using deepspeed zero3 context manager as below:
```python
with ds.zero.Init(config_dict_or_path=deepspeed):
model …
-
**Describe the bug**
4张80G的A100好像不能支持基于lora的7b bloom在batch为4的条件下训练,Colossalai是可以的,比较困惑,我对比了一下,batch只能设置到1
**To Reproduce**
下面是我稍微修改适配bloom的脚本(官方只公开适配facebook的opt脚本)
官方指出gradient_checkpointing和o…
-
### When did you clone our code?
I cloned the code base after 5/1/23
### Describe the issue
Issue: When I use deepspeed zero3 to pretrainning LLaVA-13B on 4 * A100(40G),I got an error shows below. …
-
### Describe the bug
I found that the **CPU** memory increase happens when accelerate calls "loss.backward()" (https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L198…