Segmentation fault - Githubissues

skydh commented 2 months ago

Current thread 0x00007fa75d499740 (most recent call first): File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456 in _conv_forward File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460 in forward File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541 in _call_impl File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl File "/mnt/workspace/segment_003/Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/backbone/swin_transformer.py", line 491 in forward File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541 in _call_impl File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl File "/mnt/workspace/segment_003/Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/backbone/swin_transformer.py", line 716 in forward File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541 in _call_impl File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl File "/mnt/workspace/segment_003/Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/backbone/backbone.py", line 151 in forward File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541 in _call_impl File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl File "/mnt/workspace/segment_003/Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/groundingdino.py", line 289 in forward File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541 in _call_impl File "/home/pai/envs/segment_003/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl File "/mnt/workspace/segment_003/Grounded-Segment-Anything/GroundingDINO/groundingdino/util/inference.py", line 67 in predict File "/mnt/workspace/segment_003/Grounded-Segment-Anything/001.py", line 114 in

哪个大佬能帮我看看吗

zylo117 commented 2 weeks ago

grounding dino有个_c的cuda库，是要自己编译的。如果使用时和编译时cuda版本、torch版本不一致就会崩，但是又没有明显的错误提示。你看看是不是这个问题。重新编译一下就好了

ipfans commented 2 weeks ago

@zylo117 I want to double check, because if there is a CUDA version mismatch, the compilation error will be(build with CUDA_HOME to 12.1):

        File "/home/user/proj/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 417, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (12.1) mismatches the version that was used to compile
      PyTorch (11.8). Please make sure to use the same CUDA versions.

I think this may not be the root cause of this issue. I'm also encountering this :(

zylo117 commented 2 weeks ago

@zylo117 I want to double check, because if there is a CUDA version mismatch, the compilation error will be(build with CUDA_HOME to 12.1):
        File "/home/user/proj/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 417, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (12.1) mismatches the version that was used to compile
      PyTorch (11.8). Please make sure to use the same CUDA versions.
I think this may not be the root cause of this issue. I'm also encountering this :(

well for me it was torch version mismatch，i build it with 2.2 and run with 2.3，then segmentation fault. I am just saying that if there is no python assertion，it will also cause segmentation fault

ipfans commented 2 weeks ago

@zylo117 oh, you are right. I have success after downgrade pytorch==2.1.2

Baiiiiiiiiii commented 4 days ago

Hello,

@zylo117 I want to double check, because if there is a CUDA version mismatch, the compilation error will be(build with CUDA_HOME to 12.1):
    File "/home/user/proj/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 417, in _check_cuda_version
      raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
  RuntimeError:
  The detected CUDA version (12.1) mismatches the version that was used to compile
  PyTorch (11.8). Please make sure to use the same CUDA versions.
I think this may not be the root cause of this issue. I'm also encountering this :(

well for me it was torch version mismatch，i build it with 2.2 and run with 2.3，then segmentation fault. I am just saying that if there is no python assertion，it will also cause segmentation fault

I also encountered the segmentation fault error. Could you tell me how to check the build and run versions of torch?

Thank you :)

IDEA-Research / Grounded-Segment-Anything

Segmentation fault #493