Error while trying to run test

[getterupper]

Hi, when testing InverseMatrixVT3D for $200\times200\times16$ resolution, I encounter such a problem: RuntimeError: Unknown layout

Traceback (most recent call last):
  File "/data/lix/.vscode-server/extensions/ms-python.python-2024.2.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3489, in <module>
    main()
  File "/data/lix/.vscode-server/extensions/ms-python.python-2024.2.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3482, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/data/lix/.vscode-server/extensions/ms-python.python-2024.2.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2510, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "/data/lix/.vscode-server/extensions/ms-python.python-2024.2.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2517, in _exec
    globals = pydevd_runpy.run_path(file, globals, '__main__')
  File "/data/lix/.vscode-server/extensions/ms-python.python-2024.2.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/data/lix/.vscode-server/extensions/ms-python.python-2024.2.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/data/lix/.vscode-server/extensions/ms-python.python-2024.2.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "test.py", line 151, in <module>
    main()
  File "test.py", line 147, in main
    runner.test()
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmengine/runner/runner.py", line 1823, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmengine/runner/loops.py", line 443, in run
    self.run_iter(idx, data_batch)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmengine/runner/loops.py", line 462, in run_iter
    outputs = self.runner.model.test_step(data_batch)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmengine/model/wrappers/distributed.py", line 148, in test_step
    return self.module.test_step(data)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmengine/model/base_model/base_model.py", line 145, in test_step
    return self._run_forward(data, mode='predict')  # type: ignore
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmengine/model/base_model/base_model.py", line 361, in _run_forward
    results = self(**data, mode=mode)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmdet3d/models/segmentors/base.py", line 104, in forward
    return self.predict(inputs, data_samples)
  File "/DATA_EDS2/lix/InverseMatrixVT3D/inversematrixvt3d/main.py", line 138, in predict
    occ_ori_logits = self._forward(batch_inputs,batch_data_samples)
  File "/DATA_EDS2/lix/InverseMatrixVT3D/inversematrixvt3d/main.py", line 71, in _forward
    img_feats = self.extract_feat(imgs)
  File "/DATA_EDS2/lix/InverseMatrixVT3D/inversematrixvt3d/main.py", line 52, in extract_feat
    img_feats = self.img_backbone(img)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmdet/models/backbones/resnet.py", line 643, in forward
    x = res_layer(x)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmdet/models/backbones/resnet.py", line 298, in forward
    out = _inner_forward(x)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmdet/models/backbones/resnet.py", line 275, in _inner_forward
    out = self.conv2(out)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmcv/ops/modulated_deform_conv.py", line 331, in forward
    return modulated_deform_conv2d(x, offset, mask, self.weight, self.bias,
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/torch/autograd/function.py", line 553, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/data/lix/.conda/envs/InverseMatrixVT3D/lib/python3.9/site-packages/mmcv/ops/modulated_deform_conv.py", line 149, in forward
    ext_module.modulated_deform_conv_forward(
RuntimeError: Unknown layout

I can't find a way to fix this, Thanks

[DanielMing123]

Did you followed the installation instruction and install the latest pytorch?

[getterupper]

Did you followed the installation instruction and install the latest pytorch?

Yes, the version is:

torch                     2.2.1+cu118
torchaudio                2.2.1
torchvision               0.17.1

[DanielMing123]

How about mmdetection,mmdetection3d,mmsegmentation,mmengine and mmcv. Are those packages the latest one?

[getterupper]

How about mmdetection,mmdetection3d,mmsegmentation,mmengine and mmcv. Are those packages the latest one?

Yes.

Package                   Version
------------------------- -----------
absl-py                   2.1.0
addict                    2.4.0
aliyun-python-sdk-core    2.14.0
aliyun-python-sdk-kms     2.16.2
asttokens                 2.4.1
attrs                     23.2.0
black                     24.2.0
blinker                   1.7.0
cachetools                5.3.3
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        2.0.4
click                     8.1.7
colorama                  0.4.6
comm                      0.2.1
ConfigArgParse            1.7
contourpy                 1.2.0
crcmod                    1.7
cryptography              42.0.5
cycler                    0.12.1
dash                      2.15.0
dash-core-components      2.0.0
dash-html-components      2.0.0
dash-table                5.0.0
decorator                 5.1.1
descartes                 1.1.0
exceptiongroup            1.2.0
executing                 2.0.1
fastjsonschema            2.19.1
filelock                  3.13.1
fire                      0.5.0
flake8                    7.0.0
Flask                     3.0.2
focal-loss-torch          0.1.2
fonttools                 4.49.0
fsspec                    2024.2.0
grpcio                    1.62.0
huggingface-hub           0.21.3
idna                      3.4
imageio                   2.34.0
importlib-metadata        7.0.1
importlib_resources       6.1.2
iniconfig                 2.0.0
ipython                   8.18.1
ipywidgets                8.1.2
itsdangerous              2.1.2
jedi                      0.19.1
Jinja2                    3.1.3
jmespath                  0.10.0
joblib                    1.3.2
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
jupyter_core              5.7.1
jupyterlab_widgets        3.0.10
kiwisolver                1.4.5
lazy_loader               0.3
llvmlite                  0.42.0
lyft-dataset-sdk          0.0.8
Markdown                  3.5.2
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.5.3
matplotlib-inline         0.1.6
mccabe                    0.7.0
mdurl                     0.1.2
mkl-fft                   1.3.8
mkl-random                1.2.4
mkl-service               2.4.0
mmcv                      2.1.0
mmdet                     3.2.0
mmdet3d                   1.4.0
mmengine                  0.10.3
mmsegmentation            1.2.2
model-index               0.1.11
mpmath                    1.3.0
mypy-extensions           1.0.0
nbformat                  5.9.2
nest-asyncio              1.6.0
networkx                  3.1
numba                     0.59.0
numpy                     1.26.4
nuscenes-devkit           1.1.11
nvidia-cublas-cu11        11.11.3.6
nvidia-cuda-cupti-cu11    11.8.87
nvidia-cuda-nvrtc-cu11    11.8.89
nvidia-cuda-runtime-cu11  11.8.89
nvidia-cudnn-cu11         8.7.0.84
nvidia-cufft-cu11         10.9.0.58
nvidia-curand-cu11        10.3.0.86
nvidia-cusolver-cu11      11.4.1.48
nvidia-cusparse-cu11      11.7.5.86
nvidia-nccl-cu11          2.19.3
nvidia-nvtx-cu11          11.8.86
open3d                    0.18.0
opencv-python             4.9.0.80
opendatalab               0.0.10
openmim                   0.3.9
openxlab                  0.0.34
ordered-set               4.1.0
oss2                      2.17.0
packaging                 23.2
pandas                    2.2.1
parso                     0.8.3
pathspec                  0.12.1
pexpect                   4.9.0
pillow                    10.2.0
pip                       24.0
platformdirs              4.2.0
plotly                    5.19.0
pluggy                    1.4.0
plyfile                   1.0.3
prettytable               3.10.0
prompt-toolkit            3.0.43
protobuf                  4.25.3
ptyprocess                0.7.0
pure-eval                 0.2.2
pycocotools               2.0.7
pycodestyle               2.11.1
pycparser                 2.21
pycryptodome              3.20.0
pyflakes                  3.2.0
Pygments                  2.17.2
pyparsing                 3.1.1
pyquaternion              0.9.9
pytest                    8.0.2
python-dateutil           2.8.2
pytorch-loss              0.0.0
pytz                      2023.4
PyYAML                    6.0.1
referencing               0.33.0
requests                  2.28.2
retrying                  1.3.4
rich                      13.4.2
rpds-py                   0.18.0
safetensors               0.4.2
scikit-image              0.22.0
scikit-learn              1.4.1.post1
scipy                     1.12.0
setuptools                60.2.0
Shapely                   1.8.5.post1
six                       1.16.0
stack-data                0.6.3
sympy                     1.12
tabulate                  0.9.0
tenacity                  8.2.3
tensorboard               2.16.2
tensorboard-data-server   0.7.2
termcolor                 2.4.0
terminaltables            3.1.10
threadpoolctl             3.3.0
tifffile                  2024.2.12
timm                      0.9.16
tomli                     2.0.1
torch                     2.2.1+cu118
torchaudio                2.2.1
torchvision               0.17.1
tqdm                      4.65.2
traitlets                 5.14.1
trimesh                   4.1.6
triton                    2.2.0
typing_extensions         4.9.0
tzdata                    2024.1
urllib3                   1.26.18
wcwidth                   0.2.13
Werkzeug                  3.0.1
wheel                     0.42.0
widgetsnbextension        4.0.10
yapf                      0.40.2
zipp                      3.17.0

[DanielMing123]

Did you use the test command in my instruction to evaluate the model? I have never encountered this issue before.

[getterupper]

Did you use the test command in my instruction to evaluate the model? I have never encountered this issue before.

I used bash tools/dist_test.sh configs/InverseMatrixVT3D_200_200_16.py ckpt/InverseMatrixVT3D_200.pth 8, still encountered the problem.

[DanielMing123]

Please try set enable_fix=False in the config file and run evaluation again.

[getterupper]

Please try set enable_fix=False in the config file and run evaluation again.

Perhaps it's due to the cuda version. Which version did you use?

[DanielMing123]

I used pytorch 2.2 with cuda11.8. Please make sure the Pytorch version and Cuda version are aligned. I recommend using the latest Pytorch and Cuda versions.

[getterupper]

I used pytorch 2.2 with cuda11.8. Please make sure the Pytorch version and Cuda version are aligned. I recommend using the latest Pytorch and Cuda versions.

Solved it. Many thanks!