search

SamsungLabs / tr3d

[ICIP2023] TR3D: Towards Real-Time Indoor 3D Object Detection
Other
141 stars 8 forks source link

Testing Using Pretrained Model Results in Different Performance Across Multiple Runs #12

Open YunzeMan opened 1 year ago

YunzeMan commented 1 year ago

Hi, Thanks for sharing your great work. As indicated in the title, I not only got lower performance (mAP@0.25: 0.7069, mAP@0.5: 0.5699), but also different performance across multiple runs.

These are several runs using the following command: python tools/test.py configs/tr3d/tr3d_scannet-3d-18class.py ./tr3d_scannet.pth --eval mAP (Yes, I'm using the provided pretrained model on scannet)

Run 1: (mAP@0.25: 0.7069, mAP@0.5: 0.5699) Run 2: (mAP@0.25: 0.7068, mAP@0.5: 0.5716) Run 3: (mAP@0.25: 0.7069, mAP@0.5: 0.5710)

I'm using Pytorch 1.12, CUDA 11.3, CUDNN 8.

I cannot figure out where the stochasticity may come from, especially during the evaluation (test). Could you shed some lights on the possible reasons of this scenario?

Here is one of my outputs during the testing. test_log.txt

filaPro commented 1 year ago

Hi @YunzeMan ,

We didn't meet nondeterministic issues on ScanNet. Can you please share .log file of this run? Also may be check something from torch randomness guide?

YunzeMan commented 1 year ago

These are test runs so .log was not generated. But the outputs are here:

~/conda-envs/tr3d_env/lib/python3.8/site-packages/MinkowskiEngine/init.py:36: UserWarning: The environment variable OMP_NUM_THREADS not set. MinkowskiEngine will automatically set OMP_NUM_THREADS=16. If you want to set OMP_NUM_THREADS manually, please export it on the command line before running a python script. e.g. export OMP_NUM_THREADS=12; python your_program.py. It is recommended to set it below 24. warnings.warn( ~/conda-envs/tr3d_env/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( 2023-08-12 00:46:16,066 - mmcv - INFO - backbone.conv1.kernel - torch.Size([27, 3, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.norm1.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.norm1.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.conv1.kernel - torch.Size([27, 64, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.norm1.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.norm1.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.conv2.kernel - torch.Size([27, 64, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.norm2.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.norm2.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.downsample.0.kernel - torch.Size([1, 64, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.downsample.1.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.0.downsample.1.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.1.conv1.kernel - torch.Size([27, 64, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.1.norm1.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.1.norm1.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.1.conv2.kernel - torch.Size([27, 64, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.1.norm2.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.1.norm2.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.2.conv1.kernel - torch.Size([27, 64, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.2.norm1.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.2.norm1.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.2.conv2.kernel - torch.Size([27, 64, 64]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.2.norm2.bn.weight - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer1.2.norm2.bn.bias - torch.Size([64]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.conv1.kernel - torch.Size([27, 64, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.downsample.0.kernel - torch.Size([1, 64, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,066 - mmcv - INFO - backbone.layer2.0.downsample.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.0.downsample.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.1.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.1.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.1.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.1.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.1.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.1.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.2.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.2.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.2.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.2.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.2.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.2.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.3.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.3.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.3.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.3.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.3.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer2.3.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.downsample.0.kernel - torch.Size([1, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.downsample.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.0.downsample.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.1.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.1.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.1.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.1.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.1.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.1.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.2.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.2.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.2.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,067 - mmcv - INFO - backbone.layer3.2.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.2.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.2.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.3.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.3.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.3.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.3.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.3.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,068 - mmcv - INFO - backbone.layer3.3.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,071 - mmcv - INFO - backbone.layer3.4.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,071 - mmcv - INFO - backbone.layer3.4.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,071 - mmcv - INFO - backbone.layer3.4.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,071 - mmcv - INFO - backbone.layer3.4.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,071 - mmcv - INFO - backbone.layer3.4.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,071 - mmcv - INFO - backbone.layer3.4.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer3.5.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer3.5.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer3.5.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer3.5.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer3.5.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer3.5.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.downsample.0.kernel - torch.Size([1, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.downsample.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.0.downsample.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.1.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.1.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.1.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.1.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.1.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.1.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.2.conv1.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.2.norm1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.2.norm1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.2.conv2.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in MinkResNet

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.2.norm2.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - backbone.layer4.2.norm2.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - neck.lateral_block_0.0.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in TR3DNeck

2023-08-12 00:46:16,072 - mmcv - INFO - neck.lateral_block_0.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,072 - mmcv - INFO - neck.lateral_block_0.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.out_block_0.0.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in TR3DNeck

2023-08-12 00:46:16,073 - mmcv - INFO - neck.out_block_0.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.out_block_0.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.up_block_1.0.kernel - torch.Size([27, 128, 128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.up_block_1.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.up_block_1.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.lateral_block_1.0.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in TR3DNeck

2023-08-12 00:46:16,073 - mmcv - INFO - neck.lateral_block_1.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.lateral_block_1.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.out_block_1.0.kernel - torch.Size([27, 128, 128]): Initialized by user-defined init_weights in TR3DNeck

2023-08-12 00:46:16,073 - mmcv - INFO - neck.out_block_1.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.out_block_1.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.up_block_2.0.kernel - torch.Size([27, 128, 128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.up_block_2.1.bn.weight - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - neck.up_block_2.1.bn.bias - torch.Size([128]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - head.bbox_conv.kernel - torch.Size([128, 6]): Initialized by user-defined init_weights in TR3DHead

2023-08-12 00:46:16,073 - mmcv - INFO - head.bbox_conv.bias - torch.Size([1, 6]): The value is the same before and after calling init_weights of MinkSingleStage3DDetector

2023-08-12 00:46:16,073 - mmcv - INFO - head.cls_conv.kernel - torch.Size([128, 18]): Initialized by user-defined init_weights in TR3DHead

2023-08-12 00:46:16,073 - mmcv - INFO - head.cls_conv.bias - torch.Size([1, 18]): Initialized by user-defined init_weights in TR3DHead

load checkpoint from local path: ~/tr3d/pretrained/tr3d_scannet.pth [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 312/312, 16.6 task/s, elapsed: 19s, ETA: 0s +----------------+---------+---------+---------+---------+ | classes | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 | +----------------+---------+---------+---------+---------+ | cabinet | 0.5385 | 0.9005 | 0.3653 | 0.6801 | | bed | 0.8727 | 0.9259 | 0.8225 | 0.8765 | | chair | 0.9546 | 0.9825 | 0.8984 | 0.9327 | | sofa | 0.9029 | 0.9794 | 0.8380 | 0.9381 | | table | 0.7340 | 0.8657 | 0.6460 | 0.7829 | | door | 0.6012 | 0.8929 | 0.4279 | 0.6381 | | window | 0.5190 | 0.7801 | 0.2712 | 0.4539 | | bookshelf | 0.6232 | 0.8571 | 0.5411 | 0.7662 | | picture | 0.1756 | 0.4144 | 0.1299 | 0.2568 | | counter | 0.7073 | 0.8846 | 0.3866 | 0.6346 | | desk | 0.7919 | 0.9528 | 0.6255 | 0.8504 | | curtain | 0.5779 | 0.8507 | 0.3767 | 0.5522 | | refrigerator | 0.5619 | 0.9298 | 0.5254 | 0.8421 | | showercurtrain | 0.8453 | 0.8929 | 0.5634 | 0.6786 | | toilet | 0.9978 | 1.0000 | 0.9604 | 0.9655 | | sink | 0.7842 | 0.9286 | 0.5131 | 0.6429 | | bathtub | 0.8990 | 0.9677 | 0.8118 | 0.8387 | | garbagebin | 0.6368 | 0.8509 | 0.5745 | 0.7472 | +----------------+---------+---------+---------+---------+ | Overall | 0.7069 | 0.8809 | 0.5710 | 0.7265 | +----------------+---------+---------+---------+---------+ {'cabinet_AP_0.25': 0.5385425686836243, 'bed_AP_0.25': 0.8726854920387268, 'chair_AP_0.25': 0.9546445608139038, 'sofa_AP_0.25': 0.9029216170310974, 'table_AP_0.25': 0.7340226769447327, 'door_AP_0.25': 0.6011685132980347, 'window_AP_0.25': 0.5189746022224426, 'bookshelf_AP_0.25': 0.623190701007843, 'picture_AP_0.25': 0.1755560040473938, 'counter_AP_0.25': 0.7072811722755432, 'desk_AP_0.25': 0.7918834090232849, 'curtain_AP_0.25': 0.5778701305389404, 'refrigerator_AP_0.25': 0.5618564486503601, 'showercurtrain_AP_0.25': 0.845292329788208, 'toilet_AP_0.25': 0.9978459477424622, 'sink_AP_0.25': 0.7842477560043335, 'bathtub_AP_0.25': 0.8989883661270142, 'garbagebin_AP_0.25': 0.6368109583854675, 'mAP_0.25': 0.7068769335746765, 'cabinet_rec_0.25': 0.9005376344086021, 'bed_rec_0.25': 0.9259259259259259, 'chair_rec_0.25': 0.9824561403508771, 'sofa_rec_0.25': 0.979381443298969, 'table_rec_0.25': 0.8657142857142858, 'door_rec_0.25': 0.892933618843683, 'window_rec_0.25': 0.7801418439716312, 'bookshelf_rec_0.25': 0.8571428571428571, 'picture_rec_0.25': 0.4144144144144144, 'counter_rec_0.25': 0.8846153846153846, 'desk_rec_0.25': 0.952755905511811, 'curtain_rec_0.25': 0.8507462686567164, 'refrigerator_rec_0.25': 0.9298245614035088, 'showercurtrain_rec_0.25': 0.8928571428571429, 'toilet_rec_0.25': 1.0, 'sink_rec_0.25': 0.9285714285714286, 'bathtub_rec_0.25': 0.967741935483871, 'garbagebin_rec_0.25': 0.8509433962264151, 'mAR_0.25': 0.8809280104109736, 'cabinet_AP_0.50': 0.36534473299980164, 'bed_AP_0.50': 0.8225228786468506, 'chair_AP_0.50': 0.8984019160270691, 'sofa_AP_0.50': 0.8380458950996399, 'table_AP_0.50': 0.646033763885498, 'door_AP_0.50': 0.42794933915138245, 'window_AP_0.50': 0.27116885781288147, 'bookshelf_AP_0.50': 0.5410945415496826, 'picture_AP_0.50': 0.12986953556537628, 'counter_AP_0.50': 0.38663819432258606, 'desk_AP_0.50': 0.6255454421043396, 'curtain_AP_0.50': 0.3766635060310364, 'refrigerator_AP_0.50': 0.5254307985305786, 'showercurtrain_AP_0.50': 0.5634320974349976, 'toilet_AP_0.50': 0.9603590369224548, 'sink_AP_0.50': 0.5131374597549438, 'bathtub_AP_0.50': 0.8118398785591125, 'garbagebin_AP_0.50': 0.5744631290435791, 'mAP_0.50': 0.5709967017173767, 'cabinet_rec_0.50': 0.6801075268817204, 'bed_rec_0.50': 0.8765432098765432, 'chair_rec_0.50': 0.9327485380116959, 'sofa_rec_0.50': 0.9381443298969072, 'table_rec_0.50': 0.7828571428571428, 'door_rec_0.50': 0.6381156316916489, 'window_rec_0.50': 0.45390070921985815, 'bookshelf_rec_0.50': 0.7662337662337663, 'picture_rec_0.50': 0.25675675675675674, 'counter_rec_0.50': 0.6346153846153846, 'desk_rec_0.50': 0.8503937007874016, 'curtain_rec_0.50': 0.5522388059701493, 'refrigerator_rec_0.50': 0.8421052631578947, 'showercurtrain_rec_0.50': 0.6785714285714286, 'toilet_rec_0.50': 0.9655172413793104, 'sink_rec_0.50': 0.6428571428571429, 'bathtub_rec_0.50': 0.8387096774193549, 'garbagebin_rec_0.50': 0.7471698113207547, 'mAR_0.50': 0.7265325593058256}

YunzeMan commented 1 year ago

Regarding randomness, I strictly followed your steps. I checked torch randomness guide but didn't find useful directions. Setting --deterministic=True didn't seem to help.

I also trained the model following your steps. Here is the log file. 20230812_013000.log

The mAP@0.25 and mAP@0.5 are both lower than your reported value. Have you altered the codebase or parameters a little bit without noticing it?

filaPro commented 1 year ago

No, this code for sure is able to reproduce our metrics...

May by you can try our implementation in mmdetection3d codebase?

filaPro commented 1 year ago

Btw, i think i understand this little randomness in test stage. Here in SparseTensor construction the default quantization_mode is RANDOM_SUBSAMPLE following MinkowskiEngine. Can you try with UNWEIGHTED_AVERAGE here?

YunzeMan commented 1 year ago

Thanks for pointing that out. However, after changing quantization_mode of x to UNWEIGHTED_AVERAGE, the little randomness still persists. Here are results of three separate runs:

Run 1: (mAP@0.25: 0.7068, mAP@0.5: 0.5702) Run 2: (mAP@0.25: 0.7068, mAP@0.5: 0.5697) Run 3: (mAP@0.25: 0.7069, mAP@0.5: 0.5720)

What's more frustrating than the little randomness is the marginally lower performance. But since the gap isn't very large, I can perhaps work with the current version.