divamgupta / stable-diffusion-tensorflow

Stable Diffusion in TensorFlow / Keras
Other
1.57k stars 227 forks source link

Load model without the image size #59

Open ulysses500 opened 1 year ago

ulysses500 commented 1 year ago

The current version of TF Stable Diffusion needs the image width and height to load the model.

def get_models(img_height, img_width, download_weights=True):

Indeed, the graph is built again if there is another image with a different size.

Is it possible to load the model without specifying the width and height? It would save a lot of time...

soten355 commented 1 year ago

You do need to specify the width and height, but there's no reason to do it again for every generation if nothing changes. I re-wrote the code on my repo to check if anything has changed before generating again. No need to re-compile if it's already good to go!

You can also look into saving the currently compiled model and loading that instead.

ulysses500 commented 1 year ago

No problem indeed if the images have the same size.

But what if want to use the model for many images having different sizes? I don't want to load the graph again and again.

soten355 commented 1 year ago

Unfortunately the model needs to have an image size. If your computer can handle the memory use, you could create multiple classes that house pre-compiled models of different image dimensions, but I would imagine that would max out at 2 or 3 classes on the best machines.

ulysses500 commented 1 year ago

Yes, I've thought about it: it is a very good workaround but it cannot cover all cases in a clean way and it uses additional memory :-/