search

keras-team / keras-cv

Industry-strength Computer Vision workflows with Keras
Other
1.01k stars 331 forks source link

Stable Diffusion Trainer #887

Closed gvalvano closed 2 years ago

gvalvano commented 2 years ago

Hi, thanks for this excellent work! :)

Do you plan to share a trainer for the stable diffusion model?

LukeWood commented 2 years ago

As an API probably not.

As a fine tuning tutorial - definitely!

gvalvano commented 2 years ago

Cool! I think it would be super-useful to let people understand better how to replicate these experiments. I'm looking forward to it!

tanzhenyu commented 2 years ago

Cool! I think it would be super-useful to let people understand better how to replicate these experiments. I'm looking forward to it!

I think we should provide some training script, in general all KerasCV models should have reproducible result and we aim to achieve paper-claimed SOTA metrics, though external contribution welcome as we don't have immediate plans for it.

LukeWood commented 2 years ago

@tanzhenyu I think this is probably out of scope for the package in that it would cost 800K to retrain.

Maybe useful in our fine-tuning journey, but in the meantime maybe we just keep this in a keras.io tutorial? WDYT?

tanzhenyu commented 2 years ago

@tanzhenyu I think this is probably out of scope for the package in that it would cost 800K to retrain.

Maybe useful in our fine-tuning journey, but in the meantime maybe we just keep this in a keras.io tutorial? WDYT?

We can't reproduce the exact results for Stable Diffusion, I agree. Though a training script on a smaller dataset might be reasonable ask, but we might not have the bandwidth to achieve that.

bhack commented 2 years ago

I think that we had already an initial thread on June about "foundation" size models: https://github.com/keras-team/keras-cv/issues/488#issuecomment-1154998812

Can we start a public discussion thread and involve the community? As I think it is not only a StablDiffusion only topic but it is a quite general strategy for the project.

LukeWood commented 2 years ago

@tanzhenyu I think this is probably out of scope for the package in that it would cost 800K to retrain. Maybe useful in our fine-tuning journey, but in the meantime maybe we just keep this in a keras.io tutorial? WDYT?

We can't reproduce the exact results for Stable Diffusion, I agree. Though a training script on a smaller dataset might be reasonable ask, but we might not have the bandwidth to achieve that.

Lets just host this in the tutorial itself for now then evaluate promotion to a component afterwards. I have a draft of the trainer available in: https://github.com/keras-team/keras-io/pull/1088