harubaru / waifu-diffusion

stable diffusion finetuned on weeb stuff

GNU Affero General Public License v3.0

1.94k stars 177 forks source link

[FEATURE REQUEST] Add multi-threading/processing to the Dataset Validator #52

Open chavinlo opened 1 year ago

chavinlo commented 1 year ago

It's awfully slow.

I am willing to do it if someone gives me directions

chinoll commented 1 year ago

Did you install xformers? https://github.com/huggingface/diffusers/issues/1343

chavinlo commented 1 year ago

Did you install xformers? huggingface/diffusers#1343

yes but I don't think xformers has anything to do with the validation process...

cafeai commented 1 year ago

You can skip validation, which is what I would suggest. Internally, I'm personally using a preprocessing application written in Rust. This probably isn't something you want to do in Python.

lopho commented 1 year ago

small self plug for a preprocessor written in python and fully parallel: https://github.com/lopho/parallel_dataprocessor

lopho commented 1 year ago

60 has parallel validation and migration