search

aszc-dev / ComfyUI-CoreMLSuite

A set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI workflows.
GNU General Public License v3.0
131 stars 12 forks source link

More questions/clarifications for the documentation #21

Open alessandroperilli opened 12 months ago

alessandroperilli commented 12 months ago

I'm really impressed by the quality of the documentation. I'm glad that my initial suggestion led to such clarity.

A few additional questions to address in the readme that might help clarify things:

  1. People are used to refer to Apple hardware as "MPS". What's the difference between MPS, GPU (which you mention), and ANE? Is MPS=ANE? Is MPS=GPU?
  2. Which one should people choose between ANE, GPU, and MPS? What difference does it make, if any?
  3. Most Apple users adopting ComfyUI are used to adding export PYTORCH_ENABLE_MPS_FALLBACK=1 to their launch script or as a variable in their environment. Is it still necessary? If not, how do I remove it? If yes, how does that influence the behavior of the nodes with ANE/GPU/MPS?
  4. Obviously, the least disruptive scenario is the use of the experimental Core ML Adapter node. Aside from the possible errors you already mentioned, is there a performance penalty compared to manually converting the models to mlmodelc or mlpackage formats?
  5. Does the Core ML Adapter node experimentally support SDXL? Will it ever, or is there a technical limitation?
  6. People have learned that safetensors is a safe format compared to pickles. What about mlmodelc or mlpackage formats? Can people download either from a place like Civitai and be safe? Or should they always download safetensors files and convert them on their machines?
  7. Do Core ML converted models behave in the same way as their safetensors counterparts if all the parameters of a generation remain identical? Or should people expect a different image output despite the seed and everything is identical?
  8. If people use a conversion node, does the conversion take place every time you queue a generation? If so, isn't it better to convert only once and then use a load node with the newly converted checkpoint?
  9. SDXL on ANE is marked as not supported. Will it ever be supported or are there technical reasons that prevent it?
  10. The Limitation section of the Readme mentions SD1.5 and SD2.1, but makes no mentions of SDXL. Are there other limitations people should be aware of?

You have done a massive job in converting the core ComyfUI nodes so that the Apple community can finally enjoy some decent performance. I'm very grateful for the contribution. Thank you.

aszc-dev commented 12 months ago

Thank you for your kind words and feedback. That's a very comprehensive list, I'll make sure to address these points in future updates to the documentation.