liulsg commented 1 week ago

YanWenKun commented 1 week ago

报 Allocation 很可能是显存不够报错日志咋说的？（记得用一对三反撇 ``` 把日志内容包起来）

liulsg commented 1 week ago

ComfyUI Error Report

Error Details

Node Type: [Comfy3D] Convert 3DGS to Mesh with NeRF and Marching Cubes
Exception Type: RuntimeError

Exception Message: Allocation on device

Stack Trace


File "G:\AI\ComfyUI_3D-pack\ComfyUI\execution.py", line 317, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\ComfyUI\execution.py", line 192, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)

File "G:\AI\ComfyUI_3D-pack\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\ComfyUI\custom_nodes\ComfyUI-3D-Pack\nodes.py", line 1846, in convert_gs_ply
imgs, alphas = converter.fit_nerf(training_nerf_iterations, training_nerf_resolution)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\LGM\nerf_marching_cubes_converter.py", line 180, in fit_nerf
image_gt, alpha_gt = self.render_gs(pose)
                     ^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\LGM\nerf_marching_cubes_converter.py", line 82, in render_gs
out = self.gs_renderer.render(self.gaussians.unsqueeze(0), cam_view.unsqueeze(0), cam_view_proj.unsqueeze(0), cam_pos.unsqueeze(0))
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\LGM\core\gs.py", line 75, in render
rendered_image, radii, rendered_depth, rendered_alpha = rasterizer(
                                                        ^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\diff_gaussian_rasterization\__init__.py", line 213, in forward
return rasterize_gaussians(
       ^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\diff_gaussian_rasterization\__init__.py", line 32, in rasterize_gaussians
return _RasterizeGaussians.apply(
       ^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\torch\autograd\function.py", line 574, in apply
return super().apply(*args, **kwargs)  # type: ignore[misc]
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\diff_gaussian_rasterization\__init__.py", line 92, in forward
num_rendered, color, depth, alpha, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
                                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

## System Information
- **ComfyUI Version:** v0.2.2
- **Arguments:** ComfyUI\main.py --windows-standalone-build
- **OS:** nt
- **Python Version:** 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
- **Embedded Python:** true
- **PyTorch Version:** 2.4.0+cu121
## Devices

- **Name:** cuda:0 NVIDIA GeForce RTX 2080 Ti : cudaMallocAsync
  - **Type:** cuda
  - **VRAM Total:** 23621861376
  - **VRAM Free:** 18755109594
  - **Torch VRAM Total:** 3523215360
  - **Torch VRAM Free:** 27038426

## Logs

2024-10-12 01:42:01,320 - root - INFO - Total VRAM 22528 MB, total RAM 32662 MB 2024-10-12 01:42:01,320 - root - INFO - pytorch version: 2.4.0+cu121 2024-10-12 01:42:02,610 - root - INFO - xformers version: 0.0.27.post2 2024-10-12 01:42:02,610 - root - INFO - Set vram state to: NORMAL_VRAM 2024-10-12 01:42:02,610 - root - INFO - Device: cuda:0 NVIDIA GeForce RTX 2080 Ti : cudaMallocAsync 2024-10-12 01:42:02,822 - root - INFO - Using xformers cross attention 2024-10-12 01:42:03,992 - root - INFO - [Prompt Server] web root: G:\AI\ComfyUI_3D-pack\ComfyUI\web 2024-10-12 01:42:04,998 - py.warnings - WARNING - G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\diffusers\models\transformers\transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead. deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)

2024-10-12 01:42:06,318 - py.warnings - WARNING - G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\kiui\nn__init__.py:31: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead. @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

2024-10-12 01:42:06,318 - py.warnings - WARNING - G:\AI\ComfyUI_3D-pack\python_embeded\Lib\site-packages\kiui\nn__init__.py:37: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead. @torch.cuda.amp.custom_bwd

2024-10-12 01:42:06,363 - py.warnings - WARNING - G:\AI\ComfyUI_3D-pack\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\TriplaneGaussian\utils\ops.py:38: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead. @custom_fwd(cast_inputs=torch.float32)

2024-10-12 01:42:06,363 - py.warnings - WARNING - G:\AI\ComfyUI_3D-pack\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\TriplaneGaussian\utils\ops.py:44: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead. @custom_bwd

## Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.



## Additional Context
(Please add any additional context or steps to reproduce the error here)

liulsg commented 1 week ago

我从comfyui-manage安装的和下载的您的安装包，都是这个问题。在之前所有的自带工作流都可以运行。

YanWenKun commented 1 week ago

是 diff_gaussian_rasterization 报的错，看起来 #14 很像，那个问题的解决方法是添加 TORCH_CUDA_ARCH_LIST 环境变量，指定自己显卡对应的架构版本。我在 run_nvidia_gpu.bat 已经添加了相关命令。

你先看看 TORCH_CUDA_ARCH_LIST 有没有正常启用。
如果该项正常的话就比较棘手了，diff_gaussian_rasterization 本身是位大佬读博时顺便搞的项目，其上游项目也一年多没更新了，最麻烦的是 CUDA C++ 报错信息少得可怜，如果有其他线索/日志信息，可以试试找找原因，没有的话只能从环境配置上尝试了。

liulsg commented 1 week ago

我看有人说用您修改的较低版本torch2.2.0可行，我下载试了下，运行到转换obj网格竟然需要几百T的显存（也有可能是内存）我已经在环境变量添加了TORCH_CUDA_ARCH_LIST=12.1（我安装的cuda是12.1 ,我最后说明下：之前运行完全没问题）

YanWenKun commented 1 week ago

这个arch是gpu架构(sm)版本，比如40系ada就是8.9，而非cuda版本参考： https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

------------------ 原始邮件 ------------------ 发件人: liulsg @.> 发送时间: 2024年10月12日 13:17 收件人: YanWenKun/ComfyUI-Windows-Portable @.> 抄送: YAN Wenkun @.>, Comment @.> 主题: Re: [YanWenKun/ComfyUI-Windows-Portable] 运行到转换obj节点就报错 (Issue #21)

我看有人说用您修改的较低版本torch2.2.0可行，我下载试了下，运行到转换obj网格竟然需要几百T的显存（也有可能是内存）我已经在环境变量添加了TORCH_CUDA_ARCH_LIST=12.1（我安装的cuda是12.1 ,我最后说明下：之前运行完全没问题）

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

YanWenKun commented 1 week ago

倒是还有个点，CUDA Toolkit 12.1 可能有 bug，装最新的 12.6 即可，兼容这些基于 12.1 编译的程序

------------------ 原始邮件 ------------------ 发件人: liulsg @.> 发送时间: 2024年10月12日 13:17 收件人: YanWenKun/ComfyUI-Windows-Portable @.> 抄送: YAN Wenkun @.>, Comment @.> 主题: Re: [YanWenKun/ComfyUI-Windows-Portable] 运行到转换obj节点就报错 (Issue #21)

我看有人说用您修改的较低版本torch2.2.0可行，我下载试了下，运行到转换obj网格竟然需要几百T的显存（也有可能是内存）我已经在环境变量添加了TORCH_CUDA_ARCH_LIST=12.1（我安装的cuda是12.1 ,我最后说明下：之前运行完全没问题）

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

liulsg commented 1 week ago

我的显卡是2080ti,好像最高支持cuda12.5

liulsg commented 1 week ago

@YanWenKun 经过多个torch版本以及cuda版本的反复安装，我确定问题出在diff-gaussian-rasterization上，好像建立这个轮子的时候一直编译失败，即使重新配置了环境也不行。

YanWenKun commented 1 week ago

编译失败是指它运行时编译 JIT 代码失败，还是尝试 build wheel 失败？如果是前者的话，尝试本地编译 diff-gaussian-rasterization 也许可行（代码简略，注意配置 TORCH_CUDA_ARCH_LIST=7.5）

pip wheel -w tmp_wheels git+https://github.com/ashawkey/diff-gaussian-rasterization.git

liulsg commented 5 days ago

已经解决。cuda11.8 torch2.3.0 py311 cuda121 torch py312测试都没有问题了。解决办法： 1.重装vs2022(17.9.0版本看评论更高版本可能有问题) 2.通过anaconda 重装torch 安装comfyui 3通过comfyui-manage安装3D PACK 4.通过cuda卸载diff-gaussian-rasterization然后git diff-gaussian-rasterization 并安装 5.剩下的缺什么模块通过cuda安装 6.环境变量添加najia vs路径等

liulsg commented 5 days ago

3d-pack安装diff-gaussian-rasterization 好像编译的会出问题，但是通过git源代码再安装就可以。

YanWenKun commented 5 days ago

Nice，看来 3D-Pack 作者编译的 wheel 也有些疑难兼容问题，我考虑加一个重新编译wheel的脚本供用户执行。或者单独做一个3D-Pack only包，把它那一大堆运行时下载的模型给打包进去

Message ID: @.***>

liulsg commented 5 days ago

不知道您说的是依赖库还是需要的模型，那个模型共计70G! 至于您提到的制作单独的编译脚本，真的是一个好的想法。

YanWenKun / ComfyUI-Windows-Portable

运行到转换obj节点就报错 #21

ComfyUI Error Report

Error Details

Stack Trace