error in ms_deformable_im2col_cuda: invalid device function

[Saafke]

Hi,

I successfully followed the installation instructions in INSTALL.md, namely:

conda create --name mask2former python=3.8 -y
conda activate mask2former
conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c nvidia
pip install -U opencv-python

# under your working directory
git clone git@github.com:facebookresearch/detectron2.git
cd detectron2
pip install -e .
pip install git+https://github.com/cocodataset/panopticapi.git
pip install git+https://github.com/mcordts/cityscapesScripts.git

cd ..
git clone git@github.com:facebookresearch/Mask2Former.git
cd Mask2Former
pip install -r requirements.txt
cd mask2former/modeling/pixel_decoder/ops
sh make.sh

However, when running the demo I get the following:

[02/23 09:54:12 detectron2]: Arguments: Namespace(confidence_threshold=0.5, config_file='../configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml', input=['/home/weber/Pictures/man.png'], opts=['MODEL.WEIGHTS', '/media/weber/Ubuntu2/ubuntu2/Human_Pose/code-from-source/Mask2Former/model_final_94dc52.pkl'], output=None, video_input=None, webcam=False)
[02/23 09:54:14 fvcore.common.checkpoint]: [Checkpointer] Loading from /media/weber/Ubuntu2/ubuntu2/Human_Pose/code-from-source/Mask2Former/model_final_94dc52.pkl ...
[02/23 09:54:16 fvcore.common.checkpoint]: Reading a file from 'MaskFormer Model Zoo'
Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...
/mnt/c7dd8318-a1d3-4622-a5fb-3fc2d8819579/CORSMAL/envs/detectron2/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448278899/work/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
/mnt/c7dd8318-a1d3-4622-a5fb-3fc2d8819579/CORSMAL/envs/detectron2/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448278899/work/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)
error in ms_deformable_im2col_cuda: invalid device function
error in ms_deformable_im2col_cuda: invalid device function
error in ms_deformable_im2col_cuda: invalid device function
error in ms_deformable_im2col_cuda: invalid device function
error in ms_deformable_im2col_cuda: invalid device function
error in ms_deformable_im2col_cuda: invalid device function
[02/23 09:54:17 detectron2]: /home/weber/Pictures/man.png: detected 56 instances in 1.09s

From a web search, it seems that this error occurs when the wrong CUDA is installed. However, I correctly installed cudatoolkit 11.1 from the installation procedure above. What else could be the issue?

FYI - the demo runs fine if I run it on my CPU (using the MODEL.DEVICE cpu)

[bowenc0221]

What is the version of cuda under CUDA_HOME?

[Saafke]

Hmm, if I do echo $CUDA_HOME, it prints an empty line

[BTW - I can run detectron2 succesfully in this environment]

[9p15p]

It seems that pytorch will find the potential path of CUDA_HOME itself. And if it is None, the compling will raise error. So I think it is not the key point of this error.

if CUDA_HOME is None:
    raise NotImplementedError('CUDA_HOME is None. Please set environment variable CUDA_HOME.')

[ICE-Bro]

I solved this problem referring to 4.