RuntimeError: cuda runtime error (30) : unknown error at /media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/nms.cu:103

[Xavier-Zeng]

❓ Questions and Help

Thanks for your great contributions! When I run the demo as follows,

python webcam.py --min-image-size 800

an error happened as this:

Downloading: "https://s3-us-west-2.amazonaws.com/detectron/35858933/12_2017_baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml.01_48_14.DzEQe4wC/output/train/coco_2014_train%3Acoco_2014_valminusminival/generalized_rcnn/model_final.pkl" to /home/xavier/.torch/models/_detectron_35858933_12_2017_baselines_e2e_mask_rcnn_R-50-FPN_1x.yaml.01_48_14.DzEQe4wC_output_train_coco_2014_train%3Acoco_2014_valminusminival_generalized_rcnn_model_final.pkl
100%|████████████████████████| 362433370/362433370 [05:41<00:00, 1062580.14it/s]
THCudaCheck FAIL file=/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/nms.cu line=103 error=30 : unknown error
Traceback (most recent call last):
  File "webcam.py", line 80, in <module>
    main()
  File "webcam.py", line 71, in main
    composite = coco_demo.run_on_opencv_image(img)
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/demo/predictor.py", line 169, in run_on_opencv_image
    predictions = self.compute_prediction(image)
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/demo/predictor.py", line 200, in compute_prediction
    predictions = self.model(image_list)
  File "/home/xavier/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 50, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/home/xavier/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/rpn/rpn.py", line 96, in forward
    return self._forward_test(anchors, objectness, rpn_box_regression)
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/rpn/rpn.py", line 122, in _forward_test
    boxes = self.box_selector_test(anchors, objectness, rpn_box_regression)
  File "/home/xavier/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/rpn/inference.py", line 138, in forward
    sampled_boxes.append(self.forward_for_single_feature_map(a, o, b))
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/rpn/inference.py", line 118, in forward_for_single_feature_map
    score_field="objectness",
  File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/structures/boxlist_ops.py", line 27, in boxlist_nms
    keep = _box_nms(boxes, score, nms_thresh)
RuntimeError: cuda runtime error (30) : unknown error at /media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/nms.cu:103

Is there anything I've made wrong? How could I solve this problem? Thanks for your help!

[fmassa]

Hi,

It's difficult to say what the problem is without further information.

Can you please copy and paste the output from the environment collection script from PyTorch (or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

• PyTorch Version (e.g., 1.0):
• OS (e.g., Linux):
• How you installed PyTorch (conda, pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

[Xavier-Zeng]

Thanks for you patient reply!

• Checklist are as follows
  • PyTorch Version (1.0.0a0+87b47ff):
  • OS (Ubuntu 16.04 LTS):
  • How you installed PyTorch (I install caffe2 by source as caffe2 install):
  • Build command you used (caffe2 Build from source):
  • Python version: 3.6
  • CUDA/cuDNN version: cuda 9.0, cuDNN 7.2
  • GPU models and configuration: GTX1060(6G)  I can't run webcam.py, but I can run Mask_R-CNN_demo.ipynb successfully. BUT NOW, I reinstall pytorch from conda as official step like this:

    conda install pytorch torchvision -c pytorch

    When I run webcam.py, Another errors happened as follows:

    Traceback (most recent call last):
    File "webcam.py", line 6, in <module>
    from predictor import COCODemo
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/demo/predictor.py", line 6, in <module>
    from maskrcnn_benchmark.modeling.detector import build_detection_model
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/detector/__init__.py", line 2, in <module>
    from .detectors import build_detection_model
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/detector/detectors.py", line 2, in <module>
    from .generalized_rcnn import GeneralizedRCNN
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 11, in <module>
    from ..backbone import build_backbone
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/__init__.py", line 2, in <module>
    from .backbone import build_backbone
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/backbone.py", line 7, in <module>
    from . import resnet
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/resnet.py", line 19, in <module>
    from maskrcnn_benchmark.layers import FrozenBatchNorm2d
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/layers/__init__.py", line 8, in <module>
    from .nms import nms
    File "/media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/layers/nms.py", line 3, in <module>
    from maskrcnn_benchmark import _C
    ImportError: /media/xavier/SSD256/caffe2_notebooks/maskrcnn-benchmark/maskrcnn_benchmark/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at5ErrorC1ENS_14SourceLocationERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

    And Now, I can't even run Mask_R-CNN_demo.ipynb!

[fmassa]

Hi,

Please follow the installation instructions from https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/INSTALL.md. You need to install PyTorch nightly, and you probably have conflicting libraries in your system.

I'd suggest creating a new conda environment.

[fmassa]

Also, this library runs on PyTorch, and not the caffe2 backend. There might be conflicts if you try to install and old version of caffe2 in the same environment as PyTorch.

I'm closing this issue as it seems like following the installation instructions from INSTALL.md should solve your issue, but let me know if that's not the case.

[XiaoXueShengwangrui]

Excuse me, are you solving this problem now?I have same problem as you.

Traceback (most recent call last): File "C:/Users/rrrr/Desktop/maskrcnn-benchmark-master/demo/predictor.py", line 6, in from maskrcnn_benchmark.modeling.detector import build_detection_model File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\modeling\detector__init.py", line 2, in from .detectors import build_detection_model File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\modeling\detector\detectors.py", line 2, in from .generalized_rcnn import GeneralizedRCNN File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\modeling\detector\generalized_rcnn.py", line 11, in from ..backbone import build_backbone File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\modeling\backbone__init__.py", line 2, in from .backbone import build_backbone File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\modeling\backbone\backbone.py", line 7, in from maskrcnn_benchmark.modeling.make_layers import conv_with_kaiming_uniform File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\modeling\make_layers.py", line 10, in from maskrcnn_benchmark.layers import Conv2d File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\layers\init.py", line 10, in from .nms import nms File "C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\layers\nms.py", line 3, in from maskrcnn_benchmark import _C ImportError: cannot import name '_C' from 'maskrcnn_benchmark' (C:\Users\rrrr\Desktop\maskrcnn-benchmark-master\maskrcnn_benchmark\init__.py)