search

Nerogar / OneTrainer

OneTrainer is a one-stop solution for all your stable diffusion training needs.
GNU Affero General Public License v3.0
1.68k stars 139 forks source link

[Feat]: Utilize multiple GPUs and increase thread count during image captioning #293

Open jferments opened 5 months ago

jferments commented 5 months ago

Describe your use-case.

I recently was using OneTrainer UI to auto-caption ~750k images using WD14 tagger. I have a dual GPU machine, but it was only utilizing one of them (and was running at very low VRAM/power usage). Additionally, it was barely utilizing CPU at all.

I was wondering if there is any way to make it more fully utilize CPU/GPU resources to speed up batch processing of images. I have a 3.7 million image dataset I need to do next, and it could cut many hours off my processing time to be able to use multiple GPUs.

What would you like to see as a solution?

An option to select multiple GPUs to use, as well as increasing thread count for captioning.

Have you considered alternatives? List them here.

No response

O-J1 commented 5 months ago

No support. Most developers for OT do no think its worth it and ruled it out multiple times. PR's are welcome.

jferments commented 5 months ago

Do you have links to tickets where they've rejected this idea? I didn't see any for it when I searched

yggdrasil75 commented 5 months ago

69 though I fully agree, using my 3080 and 3090 together would definitely be beneficial. especially when I want to train at 2048x2048 with better parameters.

Disty0 commented 5 months ago

You can run separate taggers at the same time. You will start to get CPU bottlenecked to 60~ images per second per tagger.

Run one part of the dataset on GPU0 and other on GPU1. And do not run anything on the CPU since your CPU will get hit pretty hard with all of those image operations.