Open YunzeMan opened 1 year ago
Hi @YunzeMan ,
We didn't meet nondeterministic issues on ScanNet. Can you please share .log
file of this run? Also may be check something from torch randomness guide?
These are test runs so .log was not generated. But the outputs are here:
~/conda-envs/tr3d_env/lib/python3.8/site-packages/MinkowskiEngine/init.py:36: UserWarning: The environment variable OMP_NUM_THREADS
not set. MinkowskiEngine will automatically set OMP_NUM_THREADS=16
. If you want to set OMP_NUM_THREADS
manually, please export it on the command line before running a python script. e.g. export OMP_NUM_THREADS=12; python your_program.py
. It is recommended to set it below 24.
warnings.warn(
~/conda-envs/tr3d_env/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
warnings.warn(
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.conv1.kernel - torch.Size([27, 3, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.norm1.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.norm1.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.conv1.kernel - torch.Size([27, 64, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.norm1.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.norm1.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.conv2.kernel - torch.Size([27, 64, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.norm2.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.norm2.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.downsample.0.kernel - torch.Size([1, 64, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.downsample.1.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.0.downsample.1.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.1.conv1.kernel - torch.Size([27, 64, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.1.norm1.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.1.norm1.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.1.conv2.kernel - torch.Size([27, 64, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.1.norm2.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.1.norm2.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.2.conv1.kernel - torch.Size([27, 64, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.2.norm1.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.2.norm1.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.2.conv2.kernel - torch.Size([27, 64, 64]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.2.norm2.bn.weight - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer1.2.norm2.bn.bias - torch.Size([64]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.conv1.kernel - torch.Size([27, 64, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.downsample.0.kernel - torch.Size([1, 64, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,066 - mmcv - INFO -
backbone.layer2.0.downsample.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.0.downsample.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.1.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.1.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.1.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.1.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.1.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.1.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.2.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.2.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.2.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.2.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.2.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.2.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.3.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.3.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.3.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.3.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.3.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer2.3.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.downsample.0.kernel - torch.Size([1, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.downsample.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.0.downsample.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.1.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.1.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.1.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.1.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.1.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.1.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.2.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.2.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.2.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,067 - mmcv - INFO -
backbone.layer3.2.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.2.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.2.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.3.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.3.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.3.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.3.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.3.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,068 - mmcv - INFO -
backbone.layer3.3.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,071 - mmcv - INFO -
backbone.layer3.4.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,071 - mmcv - INFO -
backbone.layer3.4.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,071 - mmcv - INFO -
backbone.layer3.4.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,071 - mmcv - INFO -
backbone.layer3.4.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,071 - mmcv - INFO -
backbone.layer3.4.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,071 - mmcv - INFO -
backbone.layer3.4.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer3.5.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer3.5.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer3.5.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer3.5.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer3.5.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer3.5.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.downsample.0.kernel - torch.Size([1, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.downsample.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.0.downsample.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.1.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.1.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.1.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.1.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.1.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.1.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.2.conv1.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.2.norm1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.2.norm1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.2.conv2.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in MinkResNet
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.2.norm2.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
backbone.layer4.2.norm2.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
neck.lateral_block_0.0.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in TR3DNeck
2023-08-12 00:46:16,072 - mmcv - INFO -
neck.lateral_block_0.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,072 - mmcv - INFO -
neck.lateral_block_0.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.out_block_0.0.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in TR3DNeck
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.out_block_0.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.out_block_0.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.up_block_1.0.kernel - torch.Size([27, 128, 128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.up_block_1.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.up_block_1.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.lateral_block_1.0.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in TR3DNeck
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.lateral_block_1.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.lateral_block_1.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.out_block_1.0.kernel - torch.Size([27, 128, 128]):
Initialized by user-defined init_weights
in TR3DNeck
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.out_block_1.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.out_block_1.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.up_block_2.0.kernel - torch.Size([27, 128, 128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.up_block_2.1.bn.weight - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
neck.up_block_2.1.bn.bias - torch.Size([128]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
head.bbox_conv.kernel - torch.Size([128, 6]):
Initialized by user-defined init_weights
in TR3DHead
2023-08-12 00:46:16,073 - mmcv - INFO -
head.bbox_conv.bias - torch.Size([1, 6]):
The value is the same before and after calling init_weights
of MinkSingleStage3DDetector
2023-08-12 00:46:16,073 - mmcv - INFO -
head.cls_conv.kernel - torch.Size([128, 18]):
Initialized by user-defined init_weights
in TR3DHead
2023-08-12 00:46:16,073 - mmcv - INFO -
head.cls_conv.bias - torch.Size([1, 18]):
Initialized by user-defined init_weights
in TR3DHead
load checkpoint from local path: ~/tr3d/pretrained/tr3d_scannet.pth [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 312/312, 16.6 task/s, elapsed: 19s, ETA: 0s +----------------+---------+---------+---------+---------+ | classes | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 | +----------------+---------+---------+---------+---------+ | cabinet | 0.5385 | 0.9005 | 0.3653 | 0.6801 | | bed | 0.8727 | 0.9259 | 0.8225 | 0.8765 | | chair | 0.9546 | 0.9825 | 0.8984 | 0.9327 | | sofa | 0.9029 | 0.9794 | 0.8380 | 0.9381 | | table | 0.7340 | 0.8657 | 0.6460 | 0.7829 | | door | 0.6012 | 0.8929 | 0.4279 | 0.6381 | | window | 0.5190 | 0.7801 | 0.2712 | 0.4539 | | bookshelf | 0.6232 | 0.8571 | 0.5411 | 0.7662 | | picture | 0.1756 | 0.4144 | 0.1299 | 0.2568 | | counter | 0.7073 | 0.8846 | 0.3866 | 0.6346 | | desk | 0.7919 | 0.9528 | 0.6255 | 0.8504 | | curtain | 0.5779 | 0.8507 | 0.3767 | 0.5522 | | refrigerator | 0.5619 | 0.9298 | 0.5254 | 0.8421 | | showercurtrain | 0.8453 | 0.8929 | 0.5634 | 0.6786 | | toilet | 0.9978 | 1.0000 | 0.9604 | 0.9655 | | sink | 0.7842 | 0.9286 | 0.5131 | 0.6429 | | bathtub | 0.8990 | 0.9677 | 0.8118 | 0.8387 | | garbagebin | 0.6368 | 0.8509 | 0.5745 | 0.7472 | +----------------+---------+---------+---------+---------+ | Overall | 0.7069 | 0.8809 | 0.5710 | 0.7265 | +----------------+---------+---------+---------+---------+ {'cabinet_AP_0.25': 0.5385425686836243, 'bed_AP_0.25': 0.8726854920387268, 'chair_AP_0.25': 0.9546445608139038, 'sofa_AP_0.25': 0.9029216170310974, 'table_AP_0.25': 0.7340226769447327, 'door_AP_0.25': 0.6011685132980347, 'window_AP_0.25': 0.5189746022224426, 'bookshelf_AP_0.25': 0.623190701007843, 'picture_AP_0.25': 0.1755560040473938, 'counter_AP_0.25': 0.7072811722755432, 'desk_AP_0.25': 0.7918834090232849, 'curtain_AP_0.25': 0.5778701305389404, 'refrigerator_AP_0.25': 0.5618564486503601, 'showercurtrain_AP_0.25': 0.845292329788208, 'toilet_AP_0.25': 0.9978459477424622, 'sink_AP_0.25': 0.7842477560043335, 'bathtub_AP_0.25': 0.8989883661270142, 'garbagebin_AP_0.25': 0.6368109583854675, 'mAP_0.25': 0.7068769335746765, 'cabinet_rec_0.25': 0.9005376344086021, 'bed_rec_0.25': 0.9259259259259259, 'chair_rec_0.25': 0.9824561403508771, 'sofa_rec_0.25': 0.979381443298969, 'table_rec_0.25': 0.8657142857142858, 'door_rec_0.25': 0.892933618843683, 'window_rec_0.25': 0.7801418439716312, 'bookshelf_rec_0.25': 0.8571428571428571, 'picture_rec_0.25': 0.4144144144144144, 'counter_rec_0.25': 0.8846153846153846, 'desk_rec_0.25': 0.952755905511811, 'curtain_rec_0.25': 0.8507462686567164, 'refrigerator_rec_0.25': 0.9298245614035088, 'showercurtrain_rec_0.25': 0.8928571428571429, 'toilet_rec_0.25': 1.0, 'sink_rec_0.25': 0.9285714285714286, 'bathtub_rec_0.25': 0.967741935483871, 'garbagebin_rec_0.25': 0.8509433962264151, 'mAR_0.25': 0.8809280104109736, 'cabinet_AP_0.50': 0.36534473299980164, 'bed_AP_0.50': 0.8225228786468506, 'chair_AP_0.50': 0.8984019160270691, 'sofa_AP_0.50': 0.8380458950996399, 'table_AP_0.50': 0.646033763885498, 'door_AP_0.50': 0.42794933915138245, 'window_AP_0.50': 0.27116885781288147, 'bookshelf_AP_0.50': 0.5410945415496826, 'picture_AP_0.50': 0.12986953556537628, 'counter_AP_0.50': 0.38663819432258606, 'desk_AP_0.50': 0.6255454421043396, 'curtain_AP_0.50': 0.3766635060310364, 'refrigerator_AP_0.50': 0.5254307985305786, 'showercurtrain_AP_0.50': 0.5634320974349976, 'toilet_AP_0.50': 0.9603590369224548, 'sink_AP_0.50': 0.5131374597549438, 'bathtub_AP_0.50': 0.8118398785591125, 'garbagebin_AP_0.50': 0.5744631290435791, 'mAP_0.50': 0.5709967017173767, 'cabinet_rec_0.50': 0.6801075268817204, 'bed_rec_0.50': 0.8765432098765432, 'chair_rec_0.50': 0.9327485380116959, 'sofa_rec_0.50': 0.9381443298969072, 'table_rec_0.50': 0.7828571428571428, 'door_rec_0.50': 0.6381156316916489, 'window_rec_0.50': 0.45390070921985815, 'bookshelf_rec_0.50': 0.7662337662337663, 'picture_rec_0.50': 0.25675675675675674, 'counter_rec_0.50': 0.6346153846153846, 'desk_rec_0.50': 0.8503937007874016, 'curtain_rec_0.50': 0.5522388059701493, 'refrigerator_rec_0.50': 0.8421052631578947, 'showercurtrain_rec_0.50': 0.6785714285714286, 'toilet_rec_0.50': 0.9655172413793104, 'sink_rec_0.50': 0.6428571428571429, 'bathtub_rec_0.50': 0.8387096774193549, 'garbagebin_rec_0.50': 0.7471698113207547, 'mAR_0.50': 0.7265325593058256}
Regarding randomness, I strictly followed your steps. I checked torch randomness guide but didn't find useful directions. Setting --deterministic=True
didn't seem to help.
I also trained the model following your steps. Here is the log file. 20230812_013000.log
The mAP@0.25 and mAP@0.5 are both lower than your reported value. Have you altered the codebase or parameters a little bit without noticing it?
Btw, i think i understand this little randomness in test stage. Here in SparseTensor construction the default quantization_mode
is RANDOM_SUBSAMPLE
following MinkowskiEngine. Can you try with UNWEIGHTED_AVERAGE
here?
Thanks for pointing that out. However, after changing quantization_mode
of x to UNWEIGHTED_AVERAGE
, the little randomness still persists. Here are results of three separate runs:
Run 1: (mAP@0.25: 0.7068, mAP@0.5: 0.5702) Run 2: (mAP@0.25: 0.7068, mAP@0.5: 0.5697) Run 3: (mAP@0.25: 0.7069, mAP@0.5: 0.5720)
What's more frustrating than the little randomness is the marginally lower performance. But since the gap isn't very large, I can perhaps work with the current version.
Hi, Thanks for sharing your great work. As indicated in the title, I not only got lower performance (mAP@0.25: 0.7069, mAP@0.5: 0.5699), but also different performance across multiple runs.
These are several runs using the following command:
python tools/test.py configs/tr3d/tr3d_scannet-3d-18class.py ./tr3d_scannet.pth --eval mAP
(Yes, I'm using the provided pretrained model on scannet)Run 1: (mAP@0.25: 0.7069, mAP@0.5: 0.5699) Run 2: (mAP@0.25: 0.7068, mAP@0.5: 0.5716) Run 3: (mAP@0.25: 0.7069, mAP@0.5: 0.5710)
I'm using Pytorch 1.12, CUDA 11.3, CUDNN 8.
I cannot figure out where the stochasticity may come from, especially during the evaluation (test). Could you shed some lights on the possible reasons of this scenario?
Here is one of my outputs during the testing. test_log.txt