search

aszc-dev / ComfyUI-CoreMLSuite

A set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI workflows.
GNU General Public License v3.0
131 stars 12 forks source link

Question : what should i put in controlnet model .fp16.safetensors or .safetensors ? #30

Open x4080 opened 11 months ago

x4080 commented 11 months ago

Question : what should i put in controlnet model .fp16.safetensors or .safetensors ?

The reason is I'm using fp16 lineart safetensors and it seems not affecting the result

Thanks

aszc-dev commented 11 months ago

As long as you convert the model with controlnet_support set to True, in this case the cn model shouldn't make any difference.

x4080 commented 11 months ago

@aszc-dev Oh, from huggingface I have to convert it first ? My mistake then. I'll read the documentation again

Edit : Do you mean when using LCM right ? Havent tried LCM yet, just use regular realisticvision unet, but it seems lineart not affecting it

Edit2 : using .fp16.safetensors

aszc-dev commented 11 months ago

By conversion I meant the scenario such as the one presented in this workflow. You can either use this workflow as a base and add controlnet to it, or load the converted model (.molodelc or .mlpackage) using this controlnet workflow. Either way, make sure to convert the checkpoint first (if you have converted the checkpoint with the same parameters, the converter will simply skip conversion and act as a loader).

Only model checkpoints need to be converted, but they have to be converted regardless of wheather you're applying LCM LoRA (or any other LoRA[s] to it), or not.

The controlnet model doesn't need to be converted and there is no node that supports such conversion (yet?).

If you have any other questions, please let me know :)

x4080 commented 11 months ago

@aszc-dev Thanks, yes, I successfully convert the model, and use the fp16 controlnet, and it is not affecting the generation, i'll try your workflow example which is using scribble