-
Tried to load and distribute model to devices in a layerwise way, by using deepspeed zero3 context manager as below:
```python
with ds.zero.Init(config_dict_or_path=deepspeed):
model …
-
`Traceback (most recent call last):
File "C:\ai\stable-diffusion-webui\Kohya\kohya_ss\train_network.py", line 539, in
train(args)
File "C:\ai\stable-diffusion-webui\Kohya\kohya_ss\train_ne…
-
Hi there,
I noticed another critical bug (at least from mine point of view): after LoRa training and even with `do_sample` is False, consecutive inference results in different results:
Loading t…
-
### 🐛 Describe the bug
I was tried to use offload fsdp to run the vicuna training,
however, after I run this command. the code looks like keeping still for 3-4hours and prints nothing like and th…
-
## Paper and Code
Paper: [Memory Efficient Optimizers with 4-bit States](https://arxiv.org/abs/2309.01507)
Code :
- https://github.com/thu-ml/low-bit-optimizers/blob/main/lpmm/optim/optimizer.py…
-
Cant understand, how is num_epochs calculated.
I use, for example, 16 images, 1 repeat, 5 epochs, 1 batch size, 16 grad accum steps. And i get 40 epoch.
Why batches per epoch is 32?
What? 😢
Help …
-
**Describe the bug**
4张80G的A100好像不能支持基于lora的7b bloom在batch为4的条件下训练,Colossalai是可以的,比较困惑,我对比了一下,batch只能设置到1
**To Reproduce**
下面是我稍微修改适配bloom的脚本(官方只公开适配facebook的opt脚本)
官方指出gradient_checkpointing和o…
-
I encountered an out of memory issue during model training, even though I had already reduced the elements in the instruction following dataset to just one and removed the contents of conversations:
…
-
### Description & Motivation
As far as I can see there are currently 2 modes to trigger the validation during training: 1) based on number of epochs (`check_val_every_n_epoch`) and 2) based on number…
-
### Describe the bug
I am trying to run dreambooth stable diffusion on m3 max.
However I am running into an issue because of which whenever I am trying to generate the class images for the concepts…