search

NVIDIA / Stable-Diffusion-WebUI-TensorRT

TensorRT Extension for Stable Diffusion Web UI
MIT License
1.89k stars 145 forks source link

Generate different image on txt2img sometimes when using lora #170

Open newgrit1004 opened 10 months ago

newgrit1004 commented 10 months ago

I got different generated images on the same input when using same trt engine that include LoRA.

The only difference between two images is the allocated gpu memory and reserved gpu memory.

However, according to the reference, same input in the same runtime environment will produce the same output.

After an engine has been built, except for IFillLayer, it is deterministic: providing the same input in the same runtime environment will produce the same output. Reference : https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#determinism

It is very crucial for me to reproduce the same image on the same input, so I want to know the reason why this happens.

Does allocated memory and reserved memory being used affect the output image??

contentis commented 9 months ago

Does allocated memory and reserved memory being used affect the output image?

It shouldn't... Are you using the main branch of this extension? In my testing, I got the same results when using a fixed seed - but there is always a good chance that my limited testing didn't catch something.

Is this a public LoRA I could test?

newgrit1004 commented 9 months ago

Sorry for late reply. It happend in the main branch. The lora I tested was https://civitai.com/models/11722/iu. One of my guess was that text encoder or unet weights did not change after lora injection. It means that stable-diffusion-webui loaded wrong sd-unet I guess. IU lora has text encoder layer and unet layer. I separately tested merging lora only in text encoder and unet. Then the generated images are slightly reproduced. So maybe it is a bug in stable diffusion webui.

contentis commented 8 months ago

You might have some luck with the 0.2.0-rc, which should greatly improve LoRA usability and addresses the unet and text encoder separately.

bigmover commented 3 months ago

You might have some luck with the 0.2.0-rc, which should greatly improve LoRA usability and addresses the unet and text encoder separately.

Hi @contentis Could I ask you a question that lora is saved and fused base on onnx exported before which will come to an error of mismatching in shape when switch engine，whether is there currently a way to directly fuse Lora onto the engine, like pytorch?