search

filaPro / oneformer3d

[CVPR2024] OneFormer3D: One Transformer for Unified Point Cloud Segmentation
Other
348 stars 32 forks source link

Unable to reproduce the instance segmentation result on ScanNetV2 #1

Closed ghost closed 11 months ago

ghost commented 11 months ago

First of all, Thank you very much for your great work!

When I ran your code, the evaluation result was far worse than what you reported. I am getting 74.0, 66.8, 49.3 for AP_0.25, AP_0.50 and AP from your test script, whereas I am supposed to get 86.7, 78.3, 59.3 for AP_0.25, AP_0.50 and AP.

The result in AP is 12.7 lower than your reported result, which is unreasonable. I am using the validation checkpoint(oneformer3d_1xb4_scannet.pth) and config(oneformer3d_1xb4_scannet.py) that you provided for ScanNetV2 validation set result.

Could you let me know how to fix it?

filaPro commented 11 months ago

We have just checked the checkpoint, and everything is fine. Can you please share the full log of your test run?

ghost commented 11 months ago

Sorry, I was able to reproduce your result by setting all the versions of libraries exactly the same as your Dockerfile. Before, I was using the lower version of pytorch. Closing now.

jaswanthbjk commented 11 months ago

@dev0812 Can you share the environment.yaml file. If you used conda ?

filaPro commented 11 months ago

@jaswanthbjk if it helps, i can share the ouput of conda env export from our docker image:

env.yml ``` name: base channels: - pytorch - nvidia - defaults dependencies: - _libgcc_mutex=0.1=main - _openmp_mutex=5.1=1_gnu - asttokens=2.0.5=pyhd3eb1b0_0 - backcall=0.2.0=pyhd3eb1b0_0 - beautifulsoup4=4.11.1=py310h06a4308_0 - blas=1.0=mkl - brotlipy=0.7.0=py310h7f8727e_1002 - bzip2=1.0.8=h7b6447c_0 - c-ares=1.18.1=h7f8727e_0 - ca-certificates=2022.10.11=h06a4308_0 - certifi=2022.9.24=py310h06a4308_0 - cffi=1.15.1=py310h74dc2b5_0 - chardet=4.0.0=py310h06a4308_1003 - charset-normalizer=2.0.4=pyhd3eb1b0_0 - cmake=3.22.1=h1fce559_0 - conda=22.11.1=py310h06a4308_4 - conda-build=3.23.3=py310h06a4308_0 - conda-package-handling=1.9.0=py310h5eee18b_1 - cryptography=38.0.1=py310h9ce1e76_0 - cuda=11.6.1=0 - cuda-cccl=11.6.55=hf6102b2_0 - cuda-command-line-tools=11.6.2=0 - cuda-compiler=11.6.2=0 - cuda-cudart=11.6.55=he381448_0 - cuda-cudart-dev=11.6.55=h42ad0f4_0 - cuda-cuobjdump=11.6.124=h2eeebcb_0 - cuda-cupti=11.6.124=h86345e5_0 - cuda-cuxxfilt=11.6.124=hecbf4f6_0 - cuda-driver-dev=11.6.55=0 - cuda-gdb=12.0.90=0 - cuda-libraries=11.6.1=0 - cuda-libraries-dev=11.6.1=0 - cuda-memcheck=11.8.86=0 - cuda-nsight=12.0.78=0 - cuda-nsight-compute=12.0.0=0 - cuda-nvcc=11.6.124=hbba6d2d_0 - cuda-nvdisasm=12.0.76=0 - cuda-nvml-dev=11.6.55=haa9ef22_0 - cuda-nvprof=12.0.90=0 - cuda-nvprune=11.6.124=he22ec0a_0 - cuda-nvrtc=11.6.124=h020bade_0 - cuda-nvrtc-dev=11.6.124=h249d397_0 - cuda-nvtx=11.6.124=h0630a44_0 - cuda-nvvp=12.0.90=0 - cuda-runtime=11.6.1=0 - cuda-samples=11.6.101=h8efea70_0 - cuda-sanitizer-api=12.0.90=0 - cuda-toolkit=11.6.1=0 - cuda-tools=11.6.1=0 - cuda-visual-tools=11.6.1=0 - decorator=5.1.1=pyhd3eb1b0_0 - executing=0.8.3=pyhd3eb1b0_0 - expat=2.4.9=h6a678d5_0 - ffmpeg=4.3=hf484d3e_0 - filelock=3.6.0=pyhd3eb1b0_0 - flit-core=3.6.0=pyhd3eb1b0_0 - freetype=2.12.1=h4a9f257_0 - gds-tools=1.5.0.59=0 - giflib=5.2.1=h7b6447c_0 - glob2=0.7=pyhd3eb1b0_0 - gmp=6.2.1=h295c915_3 - gnutls=3.6.15=he1e5248_0 - icu=58.2=he6710b0_3 - idna=3.4=py310h06a4308_0 - intel-openmp=2021.4.0=h06a4308_3561 - ipython=8.7.0=py310h06a4308_0 - jedi=0.18.1=py310h06a4308_1 - jpeg=9e=h7f8727e_0 - krb5=1.19.2=hac12032_0 - lame=3.100=h7b6447c_0 - lcms2=2.12=h3be6417_0 - ld_impl_linux-64=2.38=h1181459_1 - lerc=3.0=h295c915_0 - libarchive=3.6.1=hab531cd_0 - libcublas=11.9.2.110=h5e84587_0 - libcublas-dev=11.9.2.110=h5c901ab_0 - libcufft=10.7.1.112=hf425ae0_0 - libcufft-dev=10.7.1.112=ha5ce4c0_0 - libcufile=1.5.0.59=0 - libcufile-dev=1.5.0.59=0 - libcurand=10.3.1.50=0 - libcurand-dev=10.3.1.50=0 - libcurl=7.86.0=h91b91d3_0 - libcusolver=11.3.4.124=h33c3c4e_0 - libcusparse=11.7.2.124=h7538f96_0 - libcusparse-dev=11.7.2.124=hbbe9722_0 - libdeflate=1.8=h7f8727e_5 - libedit=3.1.20221030=h5eee18b_0 - libev=4.33=h7f8727e_1 - libffi=3.3=he6710b0_2 - libgcc-ng=11.2.0=h1234567_1 - libgomp=11.2.0=h1234567_1 - libiconv=1.16=h7f8727e_2 - libidn2=2.3.2=h7f8727e_0 - liblief=0.12.3=h6a678d5_0 - libnghttp2=1.46.0=hce63b2e_0 - libnpp=11.6.3.124=hd2722f0_0 - libnpp-dev=11.6.3.124=h3c42840_0 - libnvjpeg=11.6.2.124=hd473ad6_0 - libnvjpeg-dev=11.6.2.124=hb5906b9_0 - libpng=1.6.37=hbc83047_0 - libssh2=1.10.0=h8f2d780_0 - libstdcxx-ng=11.2.0=h1234567_1 - libtasn1=4.16.0=h27cfd23_0 - libtiff=4.4.0=hecacb30_2 - libunistring=0.9.10=h27cfd23_0 - libuuid=1.41.5=h5eee18b_0 - libuv=1.40.0=h7b6447c_0 - libwebp=1.2.4=h11a3e52_0 - libwebp-base=1.2.4=h5eee18b_0 - libxml2=2.9.14=h74e7548_0 - lz4-c=1.9.4=h6a678d5_0 - matplotlib-inline=0.1.6=py310h06a4308_0 - mkl=2021.4.0=h06a4308_640 - mkl-service=2.4.0=py310h7f8727e_0 - mkl_fft=1.3.1=py310hd6ae3a3_0 - mkl_random=1.2.2=py310h00e6091_0 - ncurses=6.3=h5eee18b_3 - nettle=3.7.3=hbbd107a_1 - nsight-compute=2022.4.0.15=0 - openh264=2.1.1=h4ff587b_0 - openssl=1.1.1s=h7f8727e_0 - parso=0.8.3=pyhd3eb1b0_0 - patch=2.7.6=h7b6447c_1001 - patchelf=0.15.0=h6a678d5_0 - pexpect=4.8.0=pyhd3eb1b0_3 - pickleshare=0.7.5=pyhd3eb1b0_1003 - pillow=9.3.0=py310hace64e9_0 - pip=22.3.1=py310h06a4308_0 - pkginfo=1.8.3=py310h06a4308_0 - pluggy=1.0.0=py310h06a4308_1 - prompt-toolkit=3.0.20=pyhd3eb1b0_0 - psutil=5.9.0=py310h5eee18b_0 - ptyprocess=0.7.0=pyhd3eb1b0_2 - pure_eval=0.2.2=pyhd3eb1b0_0 - py-lief=0.12.3=py310h6a678d5_0 - pycosat=0.6.4=py310h5eee18b_0 - pycparser=2.21=pyhd3eb1b0_0 - pyopenssl=22.0.0=pyhd3eb1b0_0 - pysocks=1.7.1=py310h06a4308_0 - python=3.10.8=haa1d7c7_0 - python-libarchive-c=2.9=pyhd3eb1b0_1 - pytorch=1.13.1=py3.10_cuda11.6_cudnn8.3.2_0 - pytorch-cuda=11.6=h867d48c_1 - pytorch-mutex=1.0=cuda - pytz=2022.1=py310h06a4308_0 - readline=8.2=h5eee18b_0 - requests=2.28.1=py310h06a4308_0 - rhash=1.4.1=h3c74f83_1 - ripgrep=13.0.0=hbdeaff8_0 - ruamel.yaml=0.17.21=py310h5eee18b_0 - ruamel.yaml.clib=0.2.6=py310h5eee18b_1 - setuptools=65.5.0=py310h06a4308_0 - six=1.16.0=pyhd3eb1b0_1 - soupsieve=2.3.2.post1=py310h06a4308_0 - sqlite=3.40.0=h5082296_0 - stack_data=0.2.0=pyhd3eb1b0_0 - tk=8.6.12=h1ccaba5_0 - toml=0.10.2=pyhd3eb1b0_0 - toolz=0.12.0=py310h06a4308_0 - torchtext=0.14.1=py310 - torchvision=0.14.1=py310_cu116 - tqdm=4.64.1=py310h06a4308_0 - traitlets=5.7.1=py310h06a4308_0 - typing_extensions=4.4.0=py310h06a4308_0 - urllib3=1.26.13=py310h06a4308_0 - wcwidth=0.2.5=pyhd3eb1b0_0 - wheel=0.37.1=pyhd3eb1b0_0 - xz=5.2.8=h5eee18b_0 - yaml=0.2.5=h7b6447c_0 - zlib=1.2.13=h5eee18b_0 - zstd=1.5.2=ha4553b6_0 - pip: - absl-py==2.0.0 - addict==2.4.0 - ansi2html==1.8.0 - astunparse==1.6.3 - attrs==23.1.0 - black==23.11.0 - blinker==1.7.0 - cachetools==5.3.2 - ccimport==0.4.2 - click==8.1.7 - comm==0.2.0 - configargparse==1.7 - contourpy==1.2.0 - cumm-cu116==0.4.11 - cycler==0.12.1 - dash==2.14.2 - dash-core-components==2.0.0 - dash-html-components==2.0.0 - dash-table==5.0.0 - descartes==1.1.0 - dnspython==2.2.1 - exceptiongroup==1.0.4 - expecttest==0.1.4 - fastjsonschema==2.19.0 - fire==0.5.0 - flake8==6.1.0 - flask==3.0.0 - fonttools==4.45.1 - future==0.18.2 - google-auth==2.24.0 - google-auth-oauthlib==1.1.0 - grpcio==1.59.3 - hypothesis==6.61.0 - imageio==2.33.0 - importlib-metadata==6.8.0 - iniconfig==2.0.0 - ipywidgets==8.1.1 - itsdangerous==2.1.2 - jinja2==3.1.2 - joblib==1.3.2 - jsonschema==4.20.0 - jsonschema-specifications==2023.11.2 - jupyter-core==5.5.0 - jupyterlab-widgets==3.0.9 - kiwisolver==1.4.5 - lark==1.1.8 - lazy-loader==0.3 - llvmlite==0.41.1 - lyft-dataset-sdk==0.0.8 - markdown==3.5.1 - markdown-it-py==3.0.0 - markupsafe==2.1.3 - matplotlib==3.5.3 - mccabe==0.7.0 - mdurl==0.1.2 - minkowskiengine==0.5.4 - mmcv==2.0.0 - mmdet==3.0.0 - mmdet3d==1.1.0 - mmengine==0.10.1 - mmsegmentation==1.0.0 - mpmath==1.2.1 - mypy-extensions==1.0.0 - nbformat==5.7.0 - nest-asyncio==1.5.8 - networkx==3.2.1 - ninja==1.11.1.1 - numba==0.58.1 - numpy==1.26.2 - nuscenes-devkit==1.1.11 - oauthlib==3.2.2 - open3d==0.17.0 - opencv-python==4.8.1.78 - packaging==23.2 - pandas==2.1.3 - pathspec==0.11.2 - pccm==0.4.11 - platformdirs==4.0.0 - plotly==5.18.0 - plyfile==1.0.2 - portalocker==2.8.2 - prettytable==3.9.0 - protobuf==4.23.4 - pyasn1==0.5.1 - pyasn1-modules==0.3.0 - pybind11==2.11.1 - pycocotools==2.0.7 - pycodestyle==2.11.1 - pyflakes==3.1.0 - pygments==2.17.2 - pyparsing==3.1.1 - pyquaternion==0.9.9 - pytest==7.4.3 - python-dateutil==2.8.2 - python-etcd==0.4.5 - pyyaml==6.0 - referencing==0.31.1 - requests-oauthlib==1.3.1 - retrying==1.3.4 - rich==13.7.0 - rpds-py==0.13.2 - rsa==4.9 - scikit-image==0.22.0 - scikit-learn==1.3.2 - scipy==1.11.4 - shapely==1.8.5.post1 - sortedcontainers==2.4.0 - spconv-cu116==2.3.6 - sympy==1.11.1 - tenacity==8.2.3 - tensorboard==2.15.1 - tensorboard-data-server==0.7.2 - termcolor==2.3.0 - terminaltables==3.1.10 - threadpoolctl==3.2.0 - tifffile==2023.9.26 - tomli==2.0.1 - torch-scatter==2.1.2 - torchelastic==0.2.2 - trimesh==4.0.5 - types-dataclasses==0.6.6 - tzdata==2023.3 - werkzeug==3.0.1 - widgetsnbextension==4.0.9 - yapf==0.40.2 - zipp==3.17.0 prefix: /opt/conda ```