I cannot reproduce your performance results.

[20183287]

issue [#14 ]

I cannot reproduce your performance results. So, I wonder if there is a part that needs to be set differently from the code attached to github.

Thank you.

[chenyilun95]

Hi, I think there is no big difference except for cleaning the code. There could be some performance variance during the training. Could you let me know the problem you encountered?

[20183287]

We followed the guidelines of DSGN++ equally. But we used 4090RTX for the equipment. Accordingly, the build file was modified, the import was modified by the spconv version difference, and the Voxel generator was modified. The rest have not been modified. I learned DSGN++ at 60 epoch and checked the eval performance. But it didn't meet the evaluation performance of the kitti valve set that you uploaded on github.

Can you tell me what the cause?

Thank you for your reply.

[chenyilun95]

Might I ask what performance you get and what's your running environment? The model is trained on 8 V100 (32G) GPU. The used spconv is v1.2.1 as https://github.com/xy-guo/LIGA-Stereo. I will check the code recently.

[20183287]

root@095b390b9ee0:/workspace/DSGN2# pip list Package Version Editable project location

---

absl-py 1.3.0 addict 2.4.0 apex 0.1 appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 asttokens 2.1.0 astunparse 1.6.3 attrs 22.1.0 audioread 3.0.0 backcall 0.2.0 beautifulsoup4 4.11.1 bleach 5.0.1 blis 0.7.9 cachetools 5.2.0 catalogue 2.0.8 ccimport 0.4.2 certifi 2022.9.24 cffi 1.15.1 charset-normalizer 2.1.1 click 8.1.3 cloudpickle 2.2.0 cmake 3.24.1.1 colorama 0.4.6 confection 0.0.3 contourpy 1.0.6 cuda-python 11.7.0+0.g95a2041.dirty cudf 22.8.0a0+304.g6ca81bbc78.dirty cugraph 22.8.0a0+132.g2daa31b6.dirty cuml 22.8.0a0+52.g73b8d00d0.dirty cumm 0.4.9 cupy-cuda118 11.0.0 cycler 0.11.0 cymem 2.0.7 Cython 0.29.32 dask 2022.7.1 dask-cuda 22.8.0a0+36.g9860cad dask-cudf 22.8.0a0+304.g6ca81bbc78.dirty debugpy 1.6.3 decorator 5.1.1 defusedxml 0.7.1 distributed 2022.7.1 easydict 1.10 entrypoints 0.4 exceptiongroup 1.0.4 execnet 1.9.0 executing 1.2.0 expecttest 0.1.3 fastjsonschema 2.16.2 fastrlock 0.8.1 fire 0.5.0 fonttools 4.38.0 fsspec 2022.11.0 functorch 1.13.0a0+936e930 google-auth 2.14.1 google-auth-oauthlib 0.4.6 graphsurgeon 0.4.6 grpcio 1.50.0 HeapDict 1.0.1 hypothesis 5.35.1 idna 3.4 imageio 2.28.1 importlib-metadata 5.0.0 importlib-resources 5.10.0 iniconfig 1.1.1 intel-openmp 2021.4.0 ipykernel 6.17.1 ipython 8.6.0 ipython-genutils 0.2.0 jedi 0.18.1 Jinja2 3.1.2 joblib 1.2.0 json5 0.9.10 jsonschema 4.17.0 jupyter_client 7.4.7 jupyter_core 5.0.0 jupyter-tensorboard 0.2.0 jupyterlab 2.3.2 jupyterlab-pygments 0.2.2 jupyterlab-server 1.2.0 jupytext 1.14.1 kiwisolver 1.4.4 langcodes 3.3.0 lark 1.1.5 lazy_loader 0.2 librosa 0.9.2 llvmlite 0.39.1 locket 1.0.0 Markdown 3.4.1 markdown-it-py 2.2.0 MarkupSafe 2.1.1 matplotlib 3.6.2 matplotlib-inline 0.1.6 mdit-py-plugins 0.3.1 mdurl 0.1.2 mistune 2.0.4 mkl 2021.1.1 mkl-devel 2021.1.1 mkl-include 2021.1.1 mmcv 2.0.0 mmcv-full 1.5.0 mmdet 2.22.0 /workspace/DSGN2/mmdetection-v2.22.0 mmengine 0.7.3 mock 4.0.3 model-index 0.1.11 msgpack 1.0.4 murmurhash 1.0.9 nbclient 0.7.0 nbconvert 7.2.5 nbformat 5.7.0 nest-asyncio 1.5.6 networkx 3.1 ninja 1.11.1 notebook 6.4.10 numba 0.56.4 numpy 1.22.2 nvidia-dali-cuda110 1.18.0 nvidia-pyindex 1.0.9 nvtx 0.2.5 oauthlib 3.2.2 onnx 1.12.0 opencv 4.6.0 opencv-python 4.7.0.72 openmim 0.3.7 ordered-set 4.1.0 packaging 21.3 pandas 1.4.4 pandocfilters 1.5.0 parso 0.8.3 partd 1.3.0 pathy 0.8.1 pccm 0.4.7 pcdet 0.1.0+b251c84 /workspace/DSGN2 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.0.1 pip 23.1.2 pkgutil_resolve_name 1.3.10 platformdirs 2.5.4 pluggy 1.0.0 polygraphy 0.42.1 pooch 1.6.0 portalocker 2.7.0 preshed 3.0.8 prettytable 3.5.0 prometheus-client 0.15.0 prompt-toolkit 3.0.32 protobuf 3.20.1 psutil 5.9.4 ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 8.0.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybind11 2.10.1 pycocotools 2.0.2 pycparser 2.21 pydantic 1.10.2 Pygments 2.13.0 pylibcugraph 22.8.0a0+132.g2daa31b6.dirty pynvml 11.4.1 pyparsing 3.0.9 pyrsistent 0.19.2 pytest 7.2.0 pytest-rerunfailures 10.2 pytest-shard 0.1.2 pytest-xdist 3.0.2 python-dateutil 2.8.2 python-hostlist 1.22 pytorch-quantization 2.1.2 pytz 2022.6 PyWavelets 1.4.1 PyYAML 6.0 pyzmq 24.0.1 raft 22.8.0a0+70.g9070c30.dirty regex 2022.10.31 requests 2.28.1 requests-oauthlib 1.3.1 resampy 0.4.2 rich 13.3.5 rmm 22.8.0a0+62.gf6bf047.dirty rsa 4.9 scikit-image 0.20.0 scikit-learn 0.24.2 scipy 1.9.1 Send2Trash 1.8.0 setuptools 59.5.0 six 1.16.0 smart-open 5.2.1 sortedcontainers 2.4.0 soundfile 0.11.0 soupsieve 2.3.2.post1 spacy 3.4.3 spacy-legacy 3.0.10 spacy-loggers 1.0.3 spconv 2.3.6 sphinx-glpi-theme 0.3 srsly 2.4.5 stack-data 0.6.1 tabulate 0.9.0 tbb 2021.7.1 tblib 1.7.0 tensorboard 2.9.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorboardX 2.6 tensorrt 8.5.1.7 termcolor 2.3.0 terminado 0.17.0 terminaltables 3.1.10 thinc 8.1.5 threadpoolctl 3.1.0 tifffile 2023.4.12 tinycss2 1.2.1 toml 0.10.2 tomli 2.0.1 toolz 0.12.0 torch 1.13.0a0+936e930 torch-tensorrt 1.3.0a0 torchtext 0.13.0a0+fae8e8c torchvision 0.15.0a0 tornado 6.1 tqdm 4.64.1 traitlets 5.5.0 transformer-engine 0.2.0 treelite 2.4.0 treelite-runtime 2.4.0 typer 0.7.0 typing_extensions 4.4.0 ucx-py 0.27.0a0+29.ge9e81f8 uff 0.6.9 urllib3 1.26.12 wasabi 0.10.1 wcwidth 0.2.5 webencodings 0.5.1 Werkzeug 2.2.2 wheel 0.38.4 xdoctest 1.0.2 xgboost 1.6.1 yapf 0.33.0 zict 2.2.0 zipp 3.10.0

log_train.txt

spconv v1.2.1 did not work. So I used the method below.

import spconv.pytorch as spconv  # instead import spconv

[chenyilun95]

The performance of epoch 60 generally performs worse than epochs 58 and 59. Specifically, epoch 59 achieves an AP_40 score of 66.9 and the paper reports a score of 67.4 for the best model. This discrepancy can be primarily attributed to the instability of camera-based detectors, which suffer from limited data (~3700). This paper reported the best results from the last 10 epochs to select the best model on the val set following OpenPCDet. The performance of KITTI test set could be more stable as the 2x training and test data size.

The stereo-lidar copy-paste strategy causes inconsistent train/val distribution, leading to unstable performance. You can try stopping the augmentation at the last 5 epoch like PointAugment, which also boost the performance.

[chenyilun95]

I just reran the code and achieved a 68.0 3D mAP_R40 as folllows. Remember to utilize the pre-computed road plane to ensure correct object placement (which is useful for point cloud GT sampling) when implementing the copy-paste augmentation.

bbox AP:97.8326, 90.4531, 89.3151
bev  AP:89.1795, 76.5587, 68.8715
3d   AP:84.5621, 67.6433, 63.1399
aos  AP:97.38, 89.91, 88.52
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:99.2493, 95.4293, 92.2330
bev  AP:92.0972, 76.6618, 71.4122
3d   AP:85.8496, 68.0065, 62.6326
aos  AP:98.77, 94.80, 91.34
Car AP@0.70, 0.50, 0.50:
bbox AP:97.8326, 90.4531, 89.3151
bev  AP:97.6600, 90.0519, 88.2892
3d   AP:97.5008, 89.7312, 86.7245
aos  AP:97.38, 89.91, 88.52
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:99.2493, 95.4293, 92.2330
bev  AP:99.1220, 94.3411, 89.6957
3d   AP:99.0181, 92.4383, 88.9017
aos  AP:98.77, 94.80, 91.34
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:75.9097, 67.4157, 62.5993
bev  AP:59.8060, 50.9474, 45.4352
3d   AP:50.7059, 42.4805, 37.8251
aos  AP:50.16, 44.83, 41.42
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:77.4395, 68.0995, 61.8482
bev  AP:59.7342, 49.9734, 43.9142
3d   AP:49.7692, 40.6537, 35.3628
aos  AP:48.17, 42.15, 37.77
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:75.9097, 67.4157, 62.5993
bev  AP:82.0537, 74.2953, 67.3138
3d   AP:81.7955, 73.7035, 66.7733
aos  AP:50.16, 44.83, 41.42
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:77.4395, 68.0995, 61.8482
bev  AP:84.3782, 75.4768, 67.9454
3d   AP:84.1161, 74.8142, 67.3945
aos  AP:48.17, 42.15, 37.77
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:69.3356, 51.5445, 49.6337
bev  AP:64.4544, 42.9911, 40.2629
3d   AP:62.0666, 40.5678, 38.2784
aos  AP:66.62, 48.03, 46.28
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:70.1872, 51.1259, 48.6908
bev  AP:64.7008, 40.8949, 38.2105
3d   AP:61.9592, 38.5809, 36.1267
aos  AP:67.31, 47.07, 44.76
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:69.3356, 51.5445, 49.6337
bev  AP:68.7723, 48.5508, 46.6954
3d   AP:68.7498, 48.3258, 46.5023
aos  AP:66.62, 48.03, 46.28
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:70.1872, 51.1259, 48.6908
bev  AP:69.6082, 47.6923, 45.0724
3d   AP:69.5885, 47.4151, 44.5297
aos  AP:67.31, 47.07, 44.76

[weiyangdaren]

I encountered a similar issue, I think it's due to the dsgn2.yaml file not being updated as I got a size mismatch warning during loading the pretrained model.

[chenyilun95]

This is normal as the layers 3 & 4 from ResNet-18/34 have been modified for efficiency following LIGA-stereo. You might refer to the original implementation.