打包docker后运行报错

[liliwannian]

您好，我发现代码中没有compile.sh文件，我在自己服务器上打包好镜像，别人载入我的镜像运行报错。我的cuda版本为11.3，别人的是11.0。以下是报错信息： python3 /work/OBBDetection/tools/test.py /work/OBBDetection/work_dirs/tianya_extra_mixup/faster_rcnn_orpn_r50_fpn_1x_dota10.py /work/OBBDetection/work_dirs/tianya_extra_mixup/epoch_20.pth --format-only Traceback (most recent call last): File "/work/OBBDetection/tools/test.py", line 153, in <module> main() File "/work/OBBDetection/tools/test.py", line 131, in main args.show_score_thr) File "/work/OBBDetection/mmdet/apis/test.py", line 26, in single_gpu_test result = model(return_loss=False, rescale=True, **data) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 159, in forward return self.module(*inputs[0], **kwargs[0]) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/work/OBBDetection/mmdet/core/fp16/decorators.py", line 51, in new_func return old_func(*args, **kwargs) File "/work/OBBDetection/mmdet/models/detectors/base.py", line 174, in forward return self.forward_test(img, img_metas, **kwargs) File "/work/OBBDetection/mmdet/models/detectors/base.py", line 159, in forward_test return self.aug_test(imgs, img_metas, **kwargs) File "/work/OBBDetection/mmdet/models/detectors/obb/obb_two_stage.py", line 226, in aug_test proposal_list = self.rotate_aug_test_rpn(x, img_metas) File "/work/OBBDetection/mmdet/models/detectors/obb/obb_test_mixins.py", line 10, in rotate_aug_test_rpn proposal_list = self.rpn_head.simple_test_rpn(x, img_meta) File "/work/OBBDetection/mmdet/models/dense_heads/rpn_test_mixin.py", line 37, in simple_test_rpn proposal_list = self.get_bboxes(*rpn_outs, img_metas) File "/work/OBBDetection/mmdet/core/fp16/decorators.py", line 131, in new_func return old_func(*args, **kwargs) File "/work/OBBDetection/mmdet/models/dense_heads/obb/obb_anchor_head.py", line 593, in get_bboxes scale_factor, cfg, rescale) File "/work/OBBDetection/mmdet/models/dense_heads/obb/oriented_rpn_head.py", line 173, in _get_bboxes_single _, keep = arb_batched_nms(hproposals, scores, ids, nms_cfg) File "/work/OBBDetection/mmdet/ops/nms_rotated/nms_rotated_wrapper.py", line 114, in arb_batched_nms torch.cat([bboxes_for_nms, scores[:, None]], -1), **nms_cfg_) File "/work/OBBDetection/mmdet/ops/nms/nms_wrapper.py", line 53, in nms inds = nms_ext.nms(dets_th, iou_thr) **RuntimeError: CUDA error: no kernel image is available for execution on the device** 好像是nms没有被编译到吗？希望您有空能回答一下，谢谢！

[liliwannian]

RuntimeError: CUDA error: no kernel image is available for execution on the device

[liliwannian]

这是我的Dockerfile `FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu18.04

ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && apt-get upgrade -y RUN apt-get update && apt-get install -y --no-install-recommends apt-utils RUN apt-get update && apt-get install -y wget RUN apt-get update && apt-get install -y python3.7 RUN apt-get update && apt-get install -y python3.7-distutils RUN apt-get update && apt-get install -y gcc python3.7-dev RUN apt-get update && apt-get install -y build-essential RUN apt-get update && apt-get install -y cmake RUN apt-get update && apt-get install -y curl RUN apt-get update && apt-get install -y ca-certificates RUN apt-get update && apt-get install -y libjpeg-dev RUN apt-get update && apt-get install -y libpng-dev RUN apt-get update && apt-get install -y libgl1-mesa-glx RUN apt-get update && apt-get install -y libglib2.0-0 RUN apt-get update && apt-get autoremove -y RUN apt-get update && apt-get autoclean RUN rm -rf /var/lib/apt/lists/*

RUN cd /usr/bin && rm python3 RUN cd /usr/bin && ln -s python3.7 python3 RUN cd /usr/bin && ln -s python3 python RUN cd /usr/bin && wget https://bootstrap.pypa.io/get-pip.py RUN cd /usr/bin && python3 get-pip.py

RUN pip3 install --upgrade pip RUN pip3 install torch===1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html RUN pip3 install torchvision===0.8.2+cu110 -f https://download.pytorch.org/whl/torch_stable.html

RUN mkdir /work WORKDIR /work COPY ./OBBDetection /work/OBBDetection RUN cd /work/OBBDetection/BboxToolkit/ && pip3 install -v -e . RUN cd /work/OBBDetection/ && pip3 install -r requirements.txt RUN pip3 install mmpycocotools RUN pip3 install mmcv==0.6.2 RUN cd /work/OBBDetection && python3 setup.py develop ADD ./main.py /work`

[jbwang1997]

对于docker我也不是十分熟悉，python setup.py develop过程会执行编译程序，等同于AerialDetection中的setup.py complish.sh一起运行，可以尝试在镜像中重新运行OBBDetection/setup.py进行重新编译。

在重新编译之前要删除之前编译产生的build和mmdet/ops中的*.so文件

[wang11wei]

这是由于你编译的带cuda的动态库时没有包含宿主机的显卡算力

三个解决方法：

1. 编译时添加宿主机显卡算力 首先你得知道宿主机显卡的算力，部分可以参考这个。 然后在 setup.py 中添加参数 TORCH_CUDA_ARCH_LIST。 eg: TORCH_CUDA_ARCH_LIST="3,7" 或是在 dockerfile 中 build 之前添加 ENV TORCH_CUDA_ARCH_LIST="3,7" 最好都加上啦

2. 在宿主机上重新编译 用 ENTRYPOINT ["/docker-entrypoint.sh"] 替代CMD 在 docker-entrypoint.sh 中写重新编译的指令以及启动执行指令，记得要提前删除之前编译产生的build和mmdet/ops中的*.so文件（我在有的机器上成功了 有的没成功 不知道啥问题）

3. 编译 CPU 版本的 mmdect 修改 setup.py 文件，屏蔽 "WITH_CUDA" ，或者直接屏蔽关联 .cu 文件的相关代码