search

bmaltais / kohya_ss

Apache License 2.0
9.54k stars 1.23k forks source link

3060 / 12 gb epically slow LoRA training #1504

Closed elphamale closed 1 year ago

elphamale commented 1 year ago

Hello, I have 3060 with 12 gb Vram and 64 gbs RAM and I'm trying to train a lora out of 20 images sized 1megapixel. The process loads VRAM to close to 100% and RAM close to 70%. GPU utilization is close to max and CPU is idling. Aaaaand I get this per iteration time: image I must be doing something SERIOUSLY wrong or what the hell?

I tried reducing the number of images but the lowest I got was still around 6 mins per iteration.

Here's the screencap of the settings: image

I am willing to provide any info needed. Please advise.

scrumpyman commented 1 year ago

1024 resolution and batch size 10 requires an insane amount of vram, and that's not accounting sdxl, which probably requires more too. 12gb vram can barely handle 768px batch size 2 on sd 1.5. Try batch size 1, but even then it might not work.

cerarslan commented 1 year ago

Hello, I have 3060 with 12 gb Vram and 64 gbs RAM and I'm trying to train a lora out of 20 images sized 1megapixel. The process loads VRAM to close to 100% and RAM close to 70%. GPU utilization is close to max and CPU is idling. Aaaaand I get this per iteration time: image I must be doing something SERIOUSLY wrong or what the hell?

I tried reducing the number of images but the lowest I got was still around 6 mins per iteration.

Here's the screencap of the settings: image

I am willing to provide any info needed. Please advise.

Same for me. 3000 steps took 19hours and ∼21s/it. It wasn't like this before.

cerarslan commented 1 year ago

How do you solve that sir? @elphamale

elphamale commented 1 year ago

I applied a preset (can't say which one right now because I'm not at home) from the selection in webui, selected LoCON instead of Standard and specified batch of one. That got me to around 5.5 s/it, which is far less scary than 6 or 40 mins per iteration.

I can provide the training command I got later if needed.

On Wed, Sep 13, 2023, 11:00 cerarslan @.***> wrote:

How do you solve that sir? @elphamale https://github.com/elphamale

— Reply to this email directly, view it on GitHub https://github.com/bmaltais/kohya_ss/issues/1504#issuecomment-1717135686, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4V4ZNTNJRP53UITIYHR7IDX2FRYBANCNFSM6AAAAAA4RJUVYU . You are receiving this because you were mentioned.Message ID: @.***>

cerarslan commented 1 year ago

I just figure out to solve my problem. I only use standart lora. When I clicked bf16 experimental and it solved my issue.