Closed 20183287 closed 1 year ago
Hi, I think there is no big difference except for cleaning the code. There could be some performance variance during the training. Could you let me know the problem you encountered?
We followed the guidelines of DSGN++ equally. But we used 4090RTX for the equipment. Accordingly, the build file was modified, the import was modified by the spconv version difference, and the Voxel generator was modified. The rest have not been modified. I learned DSGN++ at 60 epoch and checked the eval performance. But it didn't meet the evaluation performance of the kitti valve set that you uploaded on github.
Can you tell me what the cause?
Thank you for your reply.
Might I ask what performance you get and what's your running environment? The model is trained on 8 V100 (32G) GPU. The used spconv is v1.2.1 as https://github.com/xy-guo/LIGA-Stereo. I will check the code recently.
root@095b390b9ee0:/workspace/DSGN2# pip list Package Version Editable project location
absl-py 1.3.0 addict 2.4.0 apex 0.1 appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 asttokens 2.1.0 astunparse 1.6.3 attrs 22.1.0 audioread 3.0.0 backcall 0.2.0 beautifulsoup4 4.11.1 bleach 5.0.1 blis 0.7.9 cachetools 5.2.0 catalogue 2.0.8 ccimport 0.4.2 certifi 2022.9.24 cffi 1.15.1 charset-normalizer 2.1.1 click 8.1.3 cloudpickle 2.2.0 cmake 3.24.1.1 colorama 0.4.6 confection 0.0.3 contourpy 1.0.6 cuda-python 11.7.0+0.g95a2041.dirty cudf 22.8.0a0+304.g6ca81bbc78.dirty cugraph 22.8.0a0+132.g2daa31b6.dirty cuml 22.8.0a0+52.g73b8d00d0.dirty cumm 0.4.9 cupy-cuda118 11.0.0 cycler 0.11.0 cymem 2.0.7 Cython 0.29.32 dask 2022.7.1 dask-cuda 22.8.0a0+36.g9860cad dask-cudf 22.8.0a0+304.g6ca81bbc78.dirty debugpy 1.6.3 decorator 5.1.1 defusedxml 0.7.1 distributed 2022.7.1 easydict 1.10 entrypoints 0.4 exceptiongroup 1.0.4 execnet 1.9.0 executing 1.2.0 expecttest 0.1.3 fastjsonschema 2.16.2 fastrlock 0.8.1 fire 0.5.0 fonttools 4.38.0 fsspec 2022.11.0 functorch 1.13.0a0+936e930 google-auth 2.14.1 google-auth-oauthlib 0.4.6 graphsurgeon 0.4.6 grpcio 1.50.0 HeapDict 1.0.1 hypothesis 5.35.1 idna 3.4 imageio 2.28.1 importlib-metadata 5.0.0 importlib-resources 5.10.0 iniconfig 1.1.1 intel-openmp 2021.4.0 ipykernel 6.17.1 ipython 8.6.0 ipython-genutils 0.2.0 jedi 0.18.1 Jinja2 3.1.2 joblib 1.2.0 json5 0.9.10 jsonschema 4.17.0 jupyter_client 7.4.7 jupyter_core 5.0.0 jupyter-tensorboard 0.2.0 jupyterlab 2.3.2 jupyterlab-pygments 0.2.2 jupyterlab-server 1.2.0 jupytext 1.14.1 kiwisolver 1.4.4 langcodes 3.3.0 lark 1.1.5 lazy_loader 0.2 librosa 0.9.2 llvmlite 0.39.1 locket 1.0.0 Markdown 3.4.1 markdown-it-py 2.2.0 MarkupSafe 2.1.1 matplotlib 3.6.2 matplotlib-inline 0.1.6 mdit-py-plugins 0.3.1 mdurl 0.1.2 mistune 2.0.4 mkl 2021.1.1 mkl-devel 2021.1.1 mkl-include 2021.1.1 mmcv 2.0.0 mmcv-full 1.5.0 mmdet 2.22.0 /workspace/DSGN2/mmdetection-v2.22.0 mmengine 0.7.3 mock 4.0.3 model-index 0.1.11 msgpack 1.0.4 murmurhash 1.0.9 nbclient 0.7.0 nbconvert 7.2.5 nbformat 5.7.0 nest-asyncio 1.5.6 networkx 3.1 ninja 1.11.1 notebook 6.4.10 numba 0.56.4 numpy 1.22.2 nvidia-dali-cuda110 1.18.0 nvidia-pyindex 1.0.9 nvtx 0.2.5 oauthlib 3.2.2 onnx 1.12.0 opencv 4.6.0 opencv-python 4.7.0.72 openmim 0.3.7 ordered-set 4.1.0 packaging 21.3 pandas 1.4.4 pandocfilters 1.5.0 parso 0.8.3 partd 1.3.0 pathy 0.8.1 pccm 0.4.7 pcdet 0.1.0+b251c84 /workspace/DSGN2 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.0.1 pip 23.1.2 pkgutil_resolve_name 1.3.10 platformdirs 2.5.4 pluggy 1.0.0 polygraphy 0.42.1 pooch 1.6.0 portalocker 2.7.0 preshed 3.0.8 prettytable 3.5.0 prometheus-client 0.15.0 prompt-toolkit 3.0.32 protobuf 3.20.1 psutil 5.9.4 ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 8.0.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybind11 2.10.1 pycocotools 2.0.2 pycparser 2.21 pydantic 1.10.2 Pygments 2.13.0 pylibcugraph 22.8.0a0+132.g2daa31b6.dirty pynvml 11.4.1 pyparsing 3.0.9 pyrsistent 0.19.2 pytest 7.2.0 pytest-rerunfailures 10.2 pytest-shard 0.1.2 pytest-xdist 3.0.2 python-dateutil 2.8.2 python-hostlist 1.22 pytorch-quantization 2.1.2 pytz 2022.6 PyWavelets 1.4.1 PyYAML 6.0 pyzmq 24.0.1 raft 22.8.0a0+70.g9070c30.dirty regex 2022.10.31 requests 2.28.1 requests-oauthlib 1.3.1 resampy 0.4.2 rich 13.3.5 rmm 22.8.0a0+62.gf6bf047.dirty rsa 4.9 scikit-image 0.20.0 scikit-learn 0.24.2 scipy 1.9.1 Send2Trash 1.8.0 setuptools 59.5.0 six 1.16.0 smart-open 5.2.1 sortedcontainers 2.4.0 soundfile 0.11.0 soupsieve 2.3.2.post1 spacy 3.4.3 spacy-legacy 3.0.10 spacy-loggers 1.0.3 spconv 2.3.6 sphinx-glpi-theme 0.3 srsly 2.4.5 stack-data 0.6.1 tabulate 0.9.0 tbb 2021.7.1 tblib 1.7.0 tensorboard 2.9.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorboardX 2.6 tensorrt 8.5.1.7 termcolor 2.3.0 terminado 0.17.0 terminaltables 3.1.10 thinc 8.1.5 threadpoolctl 3.1.0 tifffile 2023.4.12 tinycss2 1.2.1 toml 0.10.2 tomli 2.0.1 toolz 0.12.0 torch 1.13.0a0+936e930 torch-tensorrt 1.3.0a0 torchtext 0.13.0a0+fae8e8c torchvision 0.15.0a0 tornado 6.1 tqdm 4.64.1 traitlets 5.5.0 transformer-engine 0.2.0 treelite 2.4.0 treelite-runtime 2.4.0 typer 0.7.0 typing_extensions 4.4.0 ucx-py 0.27.0a0+29.ge9e81f8 uff 0.6.9 urllib3 1.26.12 wasabi 0.10.1 wcwidth 0.2.5 webencodings 0.5.1 Werkzeug 2.2.2 wheel 0.38.4 xdoctest 1.0.2 xgboost 1.6.1 yapf 0.33.0 zict 2.2.0 zipp 3.10.0
spconv v1.2.1 did not work. So I used the method below.
import spconv.pytorch as spconv # instead import spconv
The performance of epoch 60 generally performs worse than epochs 58 and 59. Specifically, epoch 59 achieves an AP_40 score of 66.9 and the paper reports a score of 67.4 for the best model. This discrepancy can be primarily attributed to the instability of camera-based detectors, which suffer from limited data (~3700). This paper reported the best results from the last 10 epochs to select the best model on the val set following OpenPCDet. The performance of KITTI test set could be more stable as the 2x training and test data size.
The stereo-lidar copy-paste strategy causes inconsistent train/val distribution, leading to unstable performance. You can try stopping the augmentation at the last 5 epoch like PointAugment, which also boost the performance.
I just reran the code and achieved a 68.0 3D mAP_R40 as folllows. Remember to utilize the pre-computed road plane to ensure correct object placement (which is useful for point cloud GT sampling) when implementing the copy-paste augmentation.
bbox AP:97.8326, 90.4531, 89.3151
bev AP:89.1795, 76.5587, 68.8715
3d AP:84.5621, 67.6433, 63.1399
aos AP:97.38, 89.91, 88.52
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:99.2493, 95.4293, 92.2330
bev AP:92.0972, 76.6618, 71.4122
3d AP:85.8496, 68.0065, 62.6326
aos AP:98.77, 94.80, 91.34
Car AP@0.70, 0.50, 0.50:
bbox AP:97.8326, 90.4531, 89.3151
bev AP:97.6600, 90.0519, 88.2892
3d AP:97.5008, 89.7312, 86.7245
aos AP:97.38, 89.91, 88.52
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:99.2493, 95.4293, 92.2330
bev AP:99.1220, 94.3411, 89.6957
3d AP:99.0181, 92.4383, 88.9017
aos AP:98.77, 94.80, 91.34
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:75.9097, 67.4157, 62.5993
bev AP:59.8060, 50.9474, 45.4352
3d AP:50.7059, 42.4805, 37.8251
aos AP:50.16, 44.83, 41.42
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:77.4395, 68.0995, 61.8482
bev AP:59.7342, 49.9734, 43.9142
3d AP:49.7692, 40.6537, 35.3628
aos AP:48.17, 42.15, 37.77
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:75.9097, 67.4157, 62.5993
bev AP:82.0537, 74.2953, 67.3138
3d AP:81.7955, 73.7035, 66.7733
aos AP:50.16, 44.83, 41.42
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:77.4395, 68.0995, 61.8482
bev AP:84.3782, 75.4768, 67.9454
3d AP:84.1161, 74.8142, 67.3945
aos AP:48.17, 42.15, 37.77
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:69.3356, 51.5445, 49.6337
bev AP:64.4544, 42.9911, 40.2629
3d AP:62.0666, 40.5678, 38.2784
aos AP:66.62, 48.03, 46.28
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:70.1872, 51.1259, 48.6908
bev AP:64.7008, 40.8949, 38.2105
3d AP:61.9592, 38.5809, 36.1267
aos AP:67.31, 47.07, 44.76
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:69.3356, 51.5445, 49.6337
bev AP:68.7723, 48.5508, 46.6954
3d AP:68.7498, 48.3258, 46.5023
aos AP:66.62, 48.03, 46.28
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:70.1872, 51.1259, 48.6908
bev AP:69.6082, 47.6923, 45.0724
3d AP:69.5885, 47.4151, 44.5297
aos AP:67.31, 47.07, 44.76
I encountered a similar issue, I think it's due to the dsgn2.yaml file not being updated as I got a size mismatch warning during loading the pretrained model.
This is normal as the layers 3 & 4 from ResNet-18/34 have been modified for efficiency following LIGA-stereo. You might refer to the original implementation.
issue [#14 ]
I cannot reproduce your performance results. So, I wonder if there is a part that needs to be set differently from the code attached to github.
Thank you.