Exception with IPAdapter in ComfyUI: unexpected keyword argument 'cond_alt'

[FennelFetish]

Describe the bug

Exception when executing workflow in ComfyUI with OneDiff and IPAdapter. Happens with all the nodes: IPAdapter FaceID, IPAdapter Advanced, IPAdapter/IPAdapter Unified Loader... IPAdapter does work without OneDiff.

Your environment

ComfyUI_IPAdapter_plus commit b257aecbedeb0c1367c6f67671afdeadd6e81fef (2024-04-13)

Kubuntu 22.04 LTS AMD Ryzen 9 5950X 16-Core Processor NVIDIA GeForce RTX 4090

OneDiff git commit id

d7e448773f7d666fc3ad865c46d448532614e8df (2024-04-12)

OneFlow version info

path: ['/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow']
version: 0.9.1.dev20240411+cu121
git_commit: 931ca52
cmake_build_type: Release
rdma: True
mlir: True
enterprise: False

How To Reproduce

Import ComfyUI workflow from this image here: https://github.com/siliconflow/onediff/pull/732

The complete error message

Don't know if this line is part of the error: Failed to update obj_entity in place. Exception: Not support <function CondFunc.__new__.<locals>.<lambda> at 0x7f07a456f010> with parameters Module: <module 'onediff_comfy_nodes.modules.sd_hijack_utils' from '/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/sd_hijack_utils.py'>

!!! Exception during processing !!!
Traceback (most recent call last):
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 673, in apply_ipadapter
    return (ipadapter_execute(model.clone(), ipadapter_model, clip_vision, **ipa_args), )
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 373, in ipadapter_execute
    set_model_patch_replace(model, patch_kwargs, ("input", id))
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/sd_hijack_utils.py", line 41, in <lambda>
    lambda *args, **kwargs: self(*args, **kwargs),
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/sd_hijack_utils.py", line 57, in __call__
    return self.__sub_func(self.__orig_func, *args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/hijack_ipadapter_plus/IPAdapterPlus.py", line 38, in set_model_patch_replace_fn_of
    patch = CrossAttentionPatch_OF(**patch_kwargs)
TypeError: CrossAttentionPatch.__init__() got an unexpected keyword argument 'cond_alt'

Additional context

• When removing the 'cond_alt' key from the patch_kwargs dictionary on line 361 in ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py or
• when reverting ComfyUI_IPAdapter_plus back to commit 477217b364de427827f181cf11404bfc34181c41 (the one used in https://github.com/siliconflow/onediff/pull/732 ) I get the following error instead:

  Graph file /mnt/firlefanz/StableDiffusion/ComfyUI/input/graphs/picx_10.safetensors_BaseModel/UNetModel_36194ff481_4_f1f8a7f1ca61b188044db654a526065b05d40d2524e0d31165e22847fc11c900_0.9.1.dev20240411+cu121.graph does not exist! Generating graph.
  Building a graph for <class 'register_comfy.openaimodel.UNetModel'> ...
  [ERROR](GRAPH:OneflowGraph_0:OneflowGraph) building graph got error.
  Exception in forward: e=AssertionError('tensor.numpy() is not allowed to be called in nn.Graph.build(*args) or be called by lazy tensor.')
  Recompile oneflow module ...
  Building a graph for <class 'register_comfy.openaimodel.UNetModel'> ...
  [ERROR](GRAPH:OneflowGraph_1:OneflowGraph) building graph got error.
  0%|                                                                                                                                           | 0/20 [00:05<?, ?it/s]
  !!! Exception during processing !!!
  Traceback (most recent call last):
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/onediff/src/onediff/infer_compiler/oneflow/utils.py", line 21, in wrapper
  return func(self, *args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/onediff/src/onediff/infer_compiler/utils/graph_management_utils.py", line 88, in wrapper
  ret = func(self, *args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/onediff/src/onediff/infer_compiler/oneflow/deployable_module.py", line 99, in forward
  output = dpl_graph(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 295, in __call__
  self._compile(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 861, in _compile
  return self._dynamic_input_graph_cache._compile(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/cache.py", line 121, in _compile
  return graph._compile(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 865, in _compile
  return self._compile_new(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 884, in _compile_new
  _, eager_outputs = self.build_graph(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 1429, in build_graph
  outputs = self.__build_graph(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 1577, in __build_graph
  outputs = self.build(*lazy_args, **lazy_kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/onediff/src/onediff/infer_compiler/oneflow/graph.py", line 19, in build
  return self.model(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
  result = self.__block_forward(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
  result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/openaimodel.py", line 96, in forward
  h = forward_timestep_embed(
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 44, in forward_timestep_embed
  x = layer(x, context, transformer_options)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
  result = self.__block_forward(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
  result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/attention.py", line 130, in forward
  x = block(x, context=context[i], transformer_options=transformer_options)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
  result = self.__block_forward(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
  result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/comfy/ldm/modules/attention.py", line 460, in forward
  return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 191, in checkpoint
  return func(*inputs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/comfy/ldm/modules/attention.py", line 557, in _forward
  n = attn2_replace_patch[block_attn2](n, context_attn2, value_attn2, extra_options)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/nn/modules/module.py", line 225, in __call__
  res = self.forward(*args, **kwargs)
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/hijack_ipadapter_plus/CrossAttentionPatch.py", line 152, in forward
  if sigma <= sigma_start and sigma >= sigma_end:
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/framework/tensor.py", line 109, in is_nonzero
  value = input.numpy().item()
  File "/mnt/firlefanz/StableDiffusion/ComfyUI/venv/lib/python3.10/site-packages/oneflow/framework/tensor.py", line 388, in _numpy
  assert (
  AssertionError: tensor.numpy() is not allowed to be called in nn.Graph.build(*args) or be called by lazy tensor.

Thank you for your work on OneDiff

[ccssu]

an unexpected keyword argument 'cond_alt'

CrossAttentionPatch.init() got an unexpected keyword argument 'cond_alt' This error is due to a modification made to the CrossAttentionPatch of the plugin, specifically in:https://github.com/cubiq/ComfyUI_IPAdapter_plus/blob/0d0a7b3693baf8903fe2028ff218b557d619a93d/CrossAttentionPatch.py#L9

solve

@FennelFetish This has been addressed in pull request https://github.com/siliconflow/onediff/pull/814 and is currently being progressed for merging. You can try it out using https://github.com/siliconflow/onediff/pull/814

 ## Install
### 1. onediff 
git clone https://github.com/siliconflow/onediff.git
cd onediff 
git checkout -b dev_ipa+isntantid
pip install -e .

### 2. onediff_comfy_nodes
cp -r onediff_comfy_nodes path/to/ComfyUI/custom_nodes/