search

MrForExample / ComfyUI-3D-Pack

An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, etc.)
MIT License
2.38k stars 240 forks source link

Out of Memory on a 24G GPU Card from InstantMesh #171

Open iszihan opened 5 months ago

iszihan commented 5 months ago

Hello, I'm getting memory allocation error with every workflow that involves this instant-mesh reconstruction node. I'm running on a machine with 24G gpu memory. Should this happen? How could I fix it?

Traceback (most recent call last):
  File "/home/zling/Documents/projects/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/home/zling/Documents/projects/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/home/zling/Documents/projects/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/nodes.py", line 2272, in run_LRM
    mesh_out = lrm_model.extract_mesh(
  File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/lrm_mesh.py", line 357, in extract_mesh
    mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes)
  File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/lrm_mesh.py", line 165, in get_geometry_prediction
    sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes)
  File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/lrm_mesh.py", line 110, in get_sdf_deformation_prediction
    sdf, deformation, weight = torch.utils.checkpoint.checkpoint(
  File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/_compile.py", line 24, in inner
    return torch._dynamo.disable(fn, recursive)(*args, **kwargs)
  File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
    return fn(*args, **kwargs)
  File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/_dynamo/external_utils.py", line 36, in inner
    return fn(*args, **kwargs)
  File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 494, in checkpoint
    ret = function(*args, **kwargs)
  File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/renderer/synthesizer_mesh.py", line 132, in get_geometry_prediction
    sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices)
  File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/renderer/synthesizer_mesh.py", line 76, in get_geometry_prediction
    grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1)
torch.cuda.OutOfMemoryError: Allocation on device
SergioKingOne commented 4 months ago

HI @iszihan, same issue here, have you been able resolve it?

tcarneyjohnson commented 3 months ago

Hi all,

I'm getting the same error.

I noticed that if I try running the pipeline again immediately afterwards it completes without any issues, however. Is this the case on your end as well @SergioKingOne and @iszihan ?

Also, if you scroll to the beginning of the ComyUI console do you see a message that says: "UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']."

I noticed this message appeared immediately after installing the ComfyUI 3D Pack. I'm wondering if setting this value correctly will resolve the issue, but I don't know what to set it to.

SergioKingOne commented 3 months ago

Hi @tcarneyjohnson. That env variable is for setting the available compute capability architectures you can run. That should not be a problem.

A workaround for me was to directly run instant mesh without ComfyUI (i.e. download instant mesh repo and run it with a command).

Another possible solution is to unload the MVs model once its done its job but I could not achieve it.

KimMatt commented 2 months ago

This also happens to me on the "Triplane_Gaussian_Transformers_to_3DGS_and_3DMesh.json" workflow.

I'm running a 3080 series 24gb vram which is minimum listed req for gaussian to marching cubes. Everything else I use works but the 3d marching cubes.

It crashes on this node:

image

I get the same error: 

  File "D:\Code\ComfyUI_windows_portable\python_embeded\Lib\site-packages\diff_gaussian_rasterization\__init__.py", line 92, in forward
    num_rendered, color, depth, alpha, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
                                                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Allocation on device

I get the same warning:

"UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']."
KimMatt commented 2 months ago

Fixed this by reinstalling diff-gaussian-rasterization

pip uninstall diff_gaussian_rasterization pip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"

GPU-server commented 1 month ago

Fixed this by reinstalling diff-gaussian-rasterization

pip uninstall diff_gaussian_rasterization pip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"

using the python.exe embedded or ?? Inside the custom nodes folder? Worked for me thanks.