search

aszc-dev / ComfyUI-CoreMLSuite

A set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI workflows.
GNU General Public License v3.0
92 stars 8 forks source link

Facedetailer using the experimental adapter #25

Closed cchance27 closed 7 months ago

cchance27 commented 7 months ago

I'm running the ksampler and it generates an image fine after loras, etc, works great, but when i try to then also use the converted model in a experimental node to feed into facedetailer's model node i get back an error

TypeError: Expected shape (1, 4, 64, 64), got (1, 4, 135, 97) for input: sample

not sure if i'm using the experimental adapter incorrectly or if i need to do something to get it to work.

aszc-dev commented 7 months ago

"Expected shape" in this context means the input sample, which should be (batch size x channels x height/8 x width/8). While batch size doesn't matter, since I wrote some code to get around this, and you most likely won't find channel number other than 4, the width and height parameters are set during conversion and not much can be done about it. This means that if you convert the model for 512x512, it expects samples of size 64x64 (empty latent image set to 512x512). Any other size will give you this error. This error message comes directly from underlying library I'm using to load CoreML models. Thanks for pointing this out, I should probably add some wrapper with more helpful message, stating exactly what needs to be changed. Btw, the adapter is marked as experimental, since in many cases it doesn't behave like the models loaded with standard ComfyUI nodes. Basic sampling is fine, but more advanced stuff like LoRAs, model merge, etc. simply don't work, and probably never will. The recent addition of ModelSamplingDiscrete made me use it in one of "official" workflows, since it was the easiest way to change this parameter to LCM - there's no easy way to check whether LCM Lora has been applied to the model, so if that's the case, the user has to set it manually. I didn't want to make another "Core ML ModelSamplingDiscrete" node just for this case, so the use of adapter here is justified.

cchance27 commented 7 months ago

hmmm i'm not sure how the backend for facedetailer works, it was set to 512 with a max size of 1024 which made sense it was likely trying to stretch the latent to 512x1024 or something to do the face deetailing.. i set the size to 512 max to force it to 512x512 for the detailing... but then i got 1,4,64,46 tensor error so seems it's that somehow the internal facedetailer plugins aren't sticking to the 512x512 the model requires.

Looking through the documents, any chance the enumeratedshape with shape ranges would solved this sort of issue?

cchance27 commented 7 months ago

Apparently face detailer has a new node CoreMLDetailerHookProvider that you can hook into face detailer to restrict it's internal size to a specific shape to match your model