[Feature Request:] Add option to switch to finetuned openvino models

[ClashSAN]

Such as this: https://huggingface.co/ShadowPower/waifu-diffusion.openvino/tree/main This is the anime finetuned model. A Pokemon finedtuned model came out today, and I'm guessing more people will want to port these finetuned models to openvino.

The exact file(s) to move is unclear. If a feature to switch the models with parameters is too difficult, a readme guide for how to switch out model files would be great.

[brmarkus]

Good idea. Just add a boolean command line parameter like:

    parser.add_argument('--local', default=False, action='store_true', help='Optional. Enable use of locally available network files instead of downloading via huggingface_hub.')

And check the boolean flag to either use a local (hard-coded) path or download via huggingface_hub:

    self._text_encoder = self.core.read_model(
        "local/text_encoder.xml" if local else hf_hub_download(repo_id=model, filename="text_encoder.xml"),
        "local/text_encoder.bin" if local else hf_hub_download(repo_id=model, filename="text_encoder.bin")
    )

Then call it like this:

    python demo.py --prompt "Street-art painting of Emilia Clarke in style of Banksy, photorealism" --local

And you will get Waifu-Diffusion like this:

[jkrafczyk]

demo.py already has a parameter to specify alternative models.

You can use the waifudiffusion model with the current codebase like this:

python demo.py --model "ShadowPower/waifu-diffusion.openvino" --prompt "Street-art painting of Emilia Clarke in style of Banksy, photorealism"

[brmarkus]

demo.py already has a parameter to specify alternative models.

You can use the waifudiffusion model with the current codebase like this:

python demo.py --model "ShadowPower/waifu-diffusion.openvino" --prompt "Street-art painting of Emilia Clarke in style of Banksy, photorealism"

... but only if the model is known and reachable by huggingface-hub...

[Drake53]

Does anyone know how to convert models to IR format? The openVINO docs say you need a .pb or .pbtxt file, but I can only find the .ckpt files.

[ClashSAN]

@Drake53 you're awesome, nice fork! Have you found that information out yet? So, docs say convert from bin/pt/pth (the naming convention should mean the same) to onnx I assume bin/pt/pth models are the ones in the unet folder on huggingface. I assume most people are really stuck as there is no public repo visible to convert onnx to IR format. You may need to ask @ShadowPower so we can replicate what he did. https://github.com/ShadowPower