Precompile unet to CPU

[mak448a]

Is there any way to compile unet to CPU beforehand and not have to recompile it every launch time?

[Micraow]

I think I have found a method. The compile step is to turn the model into OpenVINO IR. By editing src/backend/openvino/pipelines.py can you cache the IR. Turn to lines like this:(in line 45,52,67,74)

ov_config={"CACHE_DIR": ""},

Input the cache directory to be used, and save the file.

Now run again, and the IR will be stored.The next time you launch, you don't have to wait!

Need to mention that the cache can be big(>4.5 GB per situation) , so prepare a disk big enough. In addition, when you use another mode (previous txt2img, now img2img),or change the size of image, it still needs to compile and save the file for later use

@rupeshs Maybe you can add this option to frontend😀

[mak448a]

@Micraow Want me to add a pull request?

[Micraow]

Wait, I find openvino still compile the unet and encoders again,even though I have set cache dir... @rupeshs said that It will compile at startup and when there is size change or model change etc, so I doubt whether it is worth of caching.

[mak448a]

Ah ok! Thanks.

[rupeshs]

As of now, this is not feasible.

[Micraow]

Hi! I find this may be of some help. They can cache the compiled model! @rupeshs

[rupeshs]

I'm not sure about that, if any of the following parameters width, height, model and number of images changed then we need to recompile the model https://github.com/rupeshs/fastsdcpu/blob/6b58a4d2cb187d3feeb4c6c80c828b2ede52ce16/src/frontend/utils.py#L7C1-L7C1.

[Micraow]

However, I think in most cases, we don't change those frequently.