Request support for Tencent HunYuanDiT model

[petercham]

Project code base address: https://github.com/Tencent/HunyuanDiT

ComfyUI supporting model node address: https://github.com/city96/ComfyUI_ExtraModels

This is the only CKPT model with the richest Chinese culture training set in the world, and it would be great to implement IPAdapter_plus support!

[petercham]

This is the most basic DiT model workflow, below I added the most basic few nodes from IPAdapter_plus and my own made to crop nodes to a specified scale.

The errors are reported as follows: ……Above omitted Resized image size torch.Size([1, 992, 992, 3]) Requested to load CLIPVisionModelProjection Loading 1 new model C:\Work\SD-ComfyUI\ComfyUI\comfy\ldm\modules\attention.py:346: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.) out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False) !!! Exception during processing!!! Error(s) in loading state_dict for Resampler: size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1664]). Traceback (most recent call last): File "C:\Work\SD-ComfyUI\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Work\SD-ComfyUI\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Work\SD-ComfyUI\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Work\SD-ComfyUI\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 711, in apply_ipadapter work_model, face_image = ipadapter_execute(work_model, ipadapter_model, clip_vision, ipa_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Work\SD-ComfyUI\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 340, in ipadapter_execute ipa = IPAdapter( ^^^^^^^^^^ File "C:\Work\SD-ComfyUI\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 70, in init self.image_proj_model.load_state_dict(ipadapter_model["image_proj"]) File "C:\Work\SD-ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 2189, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Resampler: size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1664]).

[cubiq]

shouldn't ipadapter models be trained for it? even if we find the right projection the models won't be optimized for it

maybe @xiaohu2015 can tell us

[petercham]

shouldn't ipadapter models be trained for it? even if we find the right projection the models won't be optimized for it

maybe @xiaohu2015 can tell us

You're right, I'm sure there will be one soon, in fact IPAdapter itself is owned by Tencent, there's no reason why they can't support their own CKPT

[xiaohu2015]

hi, currently it is not supported