search

Fannovel16 / comfyui_controlnet_aux

ComfyUI's ControlNet Auxiliary Preprocessors
Apache License 2.0
2.13k stars 202 forks source link

Using MeshGraphormer - Error: expected key in DispatchKeySet #213

Open lord-lethris opened 8 months ago

lord-lethris commented 8 months ago

Initially got the following error:

https://github.com/Fannovel16/comfyui_controlnet_aux/issues/172

However, I followed the code changes suggested in that thread, and now I get the following error:

(I suspect this is another DirectML issue caused by the lack of AMD support (not blaming you @Fannovel16 ))

Error occurred when executing MeshGraphormer-DepthMapPreprocessor:

new(): expected key in DispatchKeySet(CPU, CUDA, HIP, XLA, MPS, IPU, XPU, HPU, Lazy, Meta) but got: PrivateUse1

File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\comfyui_controlnet_aux\node_wrappers\mesh_graphormer.py", line 90, in execute
depth_map, mask, info = model(np_image, output_type="np", detect_resolution=resolution, mask_bbox_padding=mask_bbox_padding, seed=rand_seed)
File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\__init__.py", line 28, in __call__
depth_map, mask, info = self.pipeline.get_depth(input_image, mask_bbox_padding)
File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\pipeline.py", line 372, in get_depth
cropped_depthmap, pred_2d_keypoints = self.run_inference(graphormer_input.astype(np.uint8), self._model, self.mano_model, self.mesh_sampler, scale, int(crop_len))
File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\pipeline.py", line 244, in run_inference
pred_camera, pred_3d_joints, pred_vertices_sub, pred_vertices, hidden_states, att = Graphormer_model(batch_imgs, mano, mesh_sampler)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\custom_mesh_graphormer\modeling\bert\e2e_hand_network.py", line 33, in forward
template_vertices, template_3d_joints = mesh_model.layer(template_pose, template_betas)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\custom_manopth\manolayer.py", line 194, in forward
root_trans = th_with_zeros(torch.cat([root_rot, root_j], 2))
File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\custom_manopth\tensutils.py", line 17, in th_with_zeros
padding = tensor.new([0.0, 0.0, 0.0, 1.0])

Console output regarding error :

=> loading pretrained model E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\ckpts\hr16/ControlNet-HandRefiner-pruned\hrnetv2_w64_imagenet_pretrained.pth
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\comfyui_controlnet_aux\node_wrappers\mesh_graphormer.py", line 90, in execute
    depth_map, mask, info = model(np_image, output_type="np", detect_resolution=resolution, mask_bbox_padding=mask_bbox_padding, seed=rand_seed)
  File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\__init__.py", line 28, in __call__
    depth_map, mask, info = self.pipeline.get_depth(input_image, mask_bbox_padding)
  File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\pipeline.py", line 372, in get_depth
    cropped_depthmap, pred_2d_keypoints = self.run_inference(graphormer_input.astype(np.uint8), self._model, self.mano_model, self.mesh_sampler, scale, int(crop_len))
  File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\pipeline.py", line 244, in run_inference
    pred_camera, pred_3d_joints, pred_vertices_sub, pred_vertices, hidden_states, att = Graphormer_model(batch_imgs, mano, mesh_sampler)
  File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\custom_mesh_graphormer\modeling\bert\e2e_hand_network.py", line 33, in forward
    template_vertices, template_3d_joints = mesh_model.layer(template_pose, template_betas)
  File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\custom_manopth\manolayer.py", line 194, in forward
    root_trans = th_with_zeros(torch.cat([root_rot, root_j], 2))
  File "E:\ASSETS\AI-TOOLS\COMFY-DATA\custom_nodes\comfyui_controlnet_aux\src\custom_manopth\tensutils.py", line 17, in th_with_zeros
    padding = tensor.new([0.0, 0.0, 0.0, 1.0])
RuntimeError: new(): expected key in DispatchKeySet(CPU, CUDA, HIP, XLA, MPS, IPU, XPU, HPU, Lazy, Meta) but got: PrivateUse1

Prompt executed in 407.37 seconds
isawwxkrcv commented 8 months ago

Error occurred when executing MeshGraphormer-DepthMapPreprocessor:

"filename 'storages' not found"

File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 155, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 85, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 78, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\node_wrappers\mesh_graphormer.py", line 84, in execute model = MeshGraphormerDetector.from_pretrained().to(model_management.get_torch_device()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer__init.py", line 16, in from_pretrained pipeline = MeshGraphormerMediapipe(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\pipeline.py", line 167, in init__ state_dict = torch.load(args.resume_checkpoint) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\serialization.py", line 1028, in load return _legacy_load(opened_file, map_location, pickle_module, pickle_load_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\serialization.py", line 1231, in _legacy_load return legacy_load(f) ^^^^^^^^^^^^^^ File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\serialization.py", line 1117, in legacy_load tar.extract('storages', path=tmpdir) File "tarfile.py", line 2295, in extract File "tarfile.py", line 2302, in _get_extract_tarinfo File "tarfile.py", line 1985, in getmember

how to fix it?

Layer-norm commented 8 months ago

Yes, It seems like a DirectML issue https://github.com/microsoft/DirectML/issues/400, and this should be solved in https://github.com/pytorch/pytorch/pull/95748 Maybe you can update your pytorch and DirectML first.

lord-lethris commented 7 months ago

I have the latest Torch for ML

PS C:\Users\Lethris> pip show torch
Name: torch
Version: 2.0.0
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: d:\apps\python\python310\lib\site-packages
Requires: filelock, jinja2, networkx, sympy, typing-extensions
Required-by: accelerate, clip-interrogator, fairscale, kornia, lpips, open-clip-torch, peft, pytorch-lightning, simple-lama-inpainting, stanza, thop, timm, torch-directml, torchaudio, torchmetrics, torchsde, torchvision, ultralytics
PS C:\Users\Lethris>
Layer-norm commented 7 months ago

They didn't solve this problem at all! you can test that with very simple code below.

import torch
import torch_directml
dml = torch_directml.device()

a2 = torch.tensor([1,2,3,4]).to(dml)
b2 = a.new([2,3,4,5])

Fortunatelly, we didn't use too much tensor.new, and I changed that in another way.