How to use different models?

[Dingo64]

How to use different models? The Python implementation has lots of models to choose from.

[andreae293]

Sorry for the late reply What you are looking for is this python scripts: https://github.com/huggingface/diffusers/blob/main/scripts/convert_stable_diffusion_checkpoint_to_onnx.py usage is like this python convert_stable_diffusion_checkpoint_to_onnx.py --model_path "model.ckpt" --output_path "onnx_model" i have never tried it but be aware that you probably need to install the whole diffuser stuff before attempting to convert i will try later today

[Dingo64]

Thanks @andreae293 I installed it but it fails to convert .ckpt files. On the net they say first I should use convert_original_stable_diffusion_to_diffusers and then convert_stable_diffusion_checkpoint_to_onnx.py but it just creates a whole directory full of weird files.

[andreae293]

What kind of files are created? This repo and the original c++ repo uses 3 different files for each model (prompt encoder, diffusion model, decoder) can those files be located between these files?

[Dingo64]

I got all these files:

model_index.json vae\config.json vae\diffusion_pytorch_model.bin text_encoder\config.json text_encoder\pytorch_model.bin tokenizer\tokenizer_config.json tokenizer\special_tokens_map.json tokenizer\vocab.json tokenizer\merges.txt unet\config.json unet\diffusion_pytorch_model.bin scheduler\scheduler_config.json safety_checker\config.json safety_checker\pytorch_model.bin feature_extractor\preprocessor_config.json

[andreae293]

looks fine so far i suppose, to get things to work with this repo (or the c++/android one):

unet\diffusion_pytorch_model.bin is the diffusion model, must the renamed UNetModel-fp16.bin text_encoder\pytorch_model.binis the text prompt encoder, must be renamed FrozenCLIPEmbedder-fp16.bin vae\diffusion_pytorch_model.bin contains the decoder and must be renamed AutoencoderKL-fp16.bin

the .param files remain the original one i think your vocab.json is probably the same as vocab.txt of this repo were you interested in generating a single .onnx file?

[Dingo64]

I did that and it crashed. Then I renamed "UNetModel-fp16.bin" to "UNetModel-MHA-fp16.bin" and it didn't crash but made a completely black image. All those *.bin files are just renamed zips. They all very similar in a hex editor:

[andreae293]

so just renaming the unet model dont make it crash? so the other 2 nets get accepted? that's odd.. did you perhaps used a stable diffusion 2 model? as far as i know it uses more layer so it probably need a different UNetModel-fp16.param file maybe there is a mismatch between the expected model in UNetModel-fp16.param and the model weights in UNetModel-fp16.bin. i need to try it myself and see what is the problem, unfortunately i do not have the time these days.. i will probably check it next week meanwhile can you inspect unet\config.json and see if it is similiar to UNetModel-fp16.param?

[Dingo64]

unet\config.json is completely different.

I have a clue, maybe.

Those .bin files I listed above are from converting .ckpt using convert_original_stable_diffusion_to_diffusers.py.

Then I run on those files convert_stable_diffusion_checkpoint_to_onnx.py it gave me some .onnx files which are not what your program expects. But I tried chainner to convert those .onnx to ncnn and when I tried on AutoencoderKL-fp32.onnx file from the C++ version of your program it worked. But failed on other .onnx files.

So I believe I need ONNX to NCNN converter.

[andreae293]

i thought the 3 .bin files were the onnx models.. they are described this way in edvince repo

look at this comment: https://github.com/EdVince/Stable-Diffusion-NCNN/issues/9#issuecomment-1399739072

he clearly says to use huggingface diffusers repo to convert them..

[Dingo64]

I see nothing here that could convert to ncnn: https://github.com/huggingface/diffusers/tree/main/scripts

I tried Chainner on other models (SRGAN, ESRGAN) and it nicely converts from .onnx to ncnn giving .bin+.param. It just doesn't seem to work on Stable Diffusion models.

[Dingo64]

I also tried onnxsim and onnx2ncnn from: https://github.com/Tencent/ncnn/wiki/use-ncnn-with-pytorch-or-onnx but both failed.

[andreae293]

so after translating edvince chinese comment in the aforementioned link i realized that there is no clean way to convert from onnx to ncnn.. i dont think its worth the time.. you got any update?

In case you haven't read the translation: It should be noted that the onnx model is limited to a maximum of 2G, so be sure to export the 2G fp16 version. After getting onnx, it is basically difficult to get ncnn by calling onnx2ncnn and ncnnoptimize. There will be a lot of problems. You need to use onnxsurgeon to adjust the onnx model. After adjusting to successfully generate ncnn, the ncnn model also needs to be checked again. In fact, I took the onnx2ncnn way from the beginning, but unfortunately I didn't get it through, and I changed the pnnx export method later. The export of SD is still quite troublesome, and after the export, my code is not applicable, because I have incorporated some calculations of samplers into the model.

[sumersm7]

TL;DR there is no easy way to convert and use .ckpt and .safetensor in ncnn.