[Bug] Error while evaluating TPVFormer

bqm1111 commented 10 months ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

sys.platform: linux Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] CUDA available: True numpy_random_seed: 2147483648 GPU 0: NVIDIA GeForce RTX 4080 Laptop GPU CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.1, V11.1.74 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 PyTorch: 2.1.0+cu121 PyTorch compiling details: PyTorch built with:

GCC 9.3
C++ Version: 201703
Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 12.1
NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
CuDNN 8.9.2
Magma 2.6.1
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.16.0+cu121 OpenCV: 4.8.1 MMEngine: 0.10.1 MMDetection: 3.3.0 MMDetection3D: 1.4.0+fe25f7a spconv2.0: False

Reproduces the problem - code sample

python tools/test.py projects/TPVFormer/configs/tpvformer_8xb1-2x_nus-seg.py checkpoints/tpvformer_pretrained_fcos3d_r101_dcn.pth

Reproduces the problem - command or script

python tools/test.py projects/TPVFormer/configs/tpvformer_8xb1-2x_nus-seg.py checkpoints/tpvformer_pretrained_fcos3d_r101_dcn.pth

Reproduces the problem - error message

Traceback (most recent call last):
  File "/home/sherlock/workspace/learning/mmdet3d/tools/test.py", line 149, in <module>
    main()
  File "/home/sherlock/workspace/learning/mmdet3d/tools/test.py", line 145, in main
    runner.test()
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1823, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/runner/loops.py", line 438, in run
    metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate
    _results = metric.evaluate(size)
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate
    _metrics = self.compute_metrics(results)  # type: ignore
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmdet3d/evaluation/metrics/seg_metric.py", line 130, in compute_metrics
    ret_dict = seg_eval(
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmdet3d/evaluation/functional/seg_eval.py", line 100, in seg_eval
    pred_seg[gt_seg == ignore_index] = -1
IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed

Additional information

I added indices=2 in val_dataloader in config file to debug.

val_dataloader = dict(
    batch_size=1,
    num_workers=4,
    persistent_workers=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type=dataset_type,
        data_root=data_root,
        data_prefix=data_prefix,
        ann_file='nuscenes_infos_val.pkl',
        pipeline=val_pipeline,
        test_mode=True, indices=2))

When I look into seg_eval() function to print out some info, I got the following information

seg_pred shape = [array(1391), array(1391)]
gt_labels shape = [array([11, 11, 11, ..., 15, 15, 15], dtype=uint8), array([11, 11, 11, ..., 15, 15, 15], dtype=uint8)]
pred_seg = 1391
gt_seg = [11 11 11 ... 15 15 15]

It seems that the output of TPVFormer does not generate the expected segmentation mask. How can I fix this now?

cuge1995 commented 7 months ago

The same issue, did you solved it?

cuge1995 commented 6 months ago

Here are more output

array(3089), array(3089), array(3057), array(3089), array(3057), array(3121
), array(2865), array(2865), array(2480), array(13), array(1296), array(591), array(271), array(15), arr
ay(14), array(14), array(13), array(13), array(14), array(14), array(17), array(17), array(19), array(19
), array(16), array(15), array(16), array(18), array(18), array(19), array(20), array(19), array(8), arr
ay(10), array(12), array(13), array(19), array(21), array(22), array(534), array(14), array(18), array(1
5), array(21), array(16), array(19), array(19), array(18), array(19), array(16), array(16), array(8), ar
ray(11), array(16), array(13), array(5), array(9), array(6), array(6), array(3), array(2), array(3), arr
ay(4), array(6), array(4), array(6), array(8), array(8), array(6), array(6), array(7), array(10), array(
7), array(10), array(10), array(10), array(7), array(5), array(6), array(15), array(19), array(20), arra
y(21), array(22), array(22), array(22), array(22), array(123), array(23), array(23), array(24), array(16
), array(14), array(11), array(7), array(5), array(6), array(4), array(0), array(0), array(0), array(7),
 array(3), array(14), array(6), array(6), array(20), array(23), array(25), array(9), array(23), array(9)
, array(28), array(12), array(12), array(9), array(9), array(8), array(26), array(28), array(22), array(
22), array(22), array(22), array(22), array(22), array(22), array(22), array(22), array(22), array(25),
array(25), array(22), array(24), array(24), array(22), array(23), array(21), array(21), array(21), array
(21), array(22), array(22), array(22), array(23), array(25), array(16), array(23), array(983), array(22)
, array(93), array(1240), array(250), array(24), array(25), array(27), array(19), array(18), array(21),
array(18), array(15), array(21), array(13), array(14), array(23), array(23), array(23), array(13), array
(11), array(10), array(11), array(12), array(12), array(0), array(11), array(11), array(11), array(12),
array(14), array(16), array(16), array(16), array(16), array(16), array(15), array(18), array(17), array
(18), array(12), array(18), array(19), array(12), array(18), array(20), array(20), array(21), array(22),
 array(20), array(11), array(11), array(3416), array(3453), array(3443), array(3507), array(3827), array
(4497), array(104), array(127), array(3604), array(1414), array(2973), array(2119), array(4299), array(1
3), array(35), array(448), array(21), array(2742), array(306), array(22), array(18), array(18), array(16
), array(30), array(23), array(19), array(14), array(5), array(10), array(6), array(6), array(5), array(
5), array(9), array(5), array(8), array(9), array(9), array(8), array(9), array(5055), array(223), array
(637), array(5373), array(4765), array(2396), array(858), array(347), array(951), array(636), array(247)
, array(18), array(5), array(8), array(14), array(1234), array(17), array(17), array(17), array(17), arr
ay(17), array(17), array(18), array(17), array(17), array(17), array(17), array(17), array(18), array(17
), array(18), array(20), array(19), array(19), array(19), array(20), array(20), array(20), array(20), ar
ray(20), array(20), array(350), array(30), array(30), array(29), array(30), array(279), array(23), array
(20), array(20), array(20), array(21), array(20), array(20), array(20), array(23), array(24), array(21),
 array(19), array(19), array(19), array(1273), array(21), array(2943), array(1919), array(863), array(95
), array(30), array(29), array(29), array(30), array(20), array(16), array(20), array(4046), array(1227)

cuge1995 commented 6 months ago

update

        for i in range(len(seg_logits_list)):
            seg_logits = seg_logits_list[i]
            seg_pred = seg_logits
            # print(seg_logits.size())
            # seg_pred = seg_logits.argmax(dim=0)
            # print('the seg 2222pred is',seg_pred)
            batch_data_samples[i].set_data({
                'pts_seg_logits':
                PointData(**{'pts_seg_logits': seg_logits}),
                'pred_pts_seg':
                PointData(**{'pts_semantic_mask': seg_pred})
            })
        return batch_data_samples

It works!

russellyq commented 1 month ago

Hi, @cuge1995

Can you provide full code on seg_eval.py

Emadona commented 1 month ago

Hi, @cuge1995

Can you provide full code on seg_eval.py

Hi @russellyq Did you fix the issue?

open-mmlab / mmdetection3d