Issue with KITTI "test set" evaluation turning all 0 values

sinatayebati commented 5 months ago

I've successfully trained and evaluated several models using "kitti_infos_train.pkl" and "kitti_infos_val.pkl" without any problems. However, when I attempted to test using "kitti_infos_test.pkl," all the returned recall values are showing as 0!

Below, I've provided my configuration and experiment details. I would greatly appreciate it if someone could assist me in troubleshooting this issue. Thank you.

Openpcdet version: pcdet+0.6.0

Testing model: PVRCNN

bash scripts/dist_test.sh 2 --cfg_file cfgs/kitti_models/pv_rcnn.yaml --ckpt ../output/kitti_models/pv_rcnn/default/ckpt/checkpoint_epoch_78.pth

Evaluation result: return _VF.meshgrid(tensors, kwargs) # type: ignore[attr-defined] 2024-04-06 17:25:30,613 INFO ==> Loading parameters from checkpoint ../output/kitti_models/pv_rcnn_09_final/default/ckpt/checkpoint_epoch_78.pth to GPU 2024-04-06 17:25:30,695 INFO ==> Checkpoint trained from version: pcdet+0.6.0+255db8f+pybd06a67 2024-04-06 17:25:30,721 INFO ==> Done (loaded 367/367) 2024-04-06 17:25:30,740 INFO EPOCH 78 EVALUATION eval: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3759/3759 [07:27<00:00, 8.40it/s, recall_0.3=(0, 0) / 0] 2024-04-06 17:32:58,430 INFO * Performance of EPOCH 78 *** 2024-04-06 17:32:58,430 INFO Generate label finished(sec_per_example: 0.0595 second). 2024-04-06 17:32:58,430 INFO recall_roi_0.3: 0.000000 2024-04-06 17:32:58,430 INFO recall_rcnn_0.3: 0.000000 2024-04-06 17:32:58,430 INFO recall_roi_0.5: 0.000000 2024-04-06 17:32:58,430 INFO recall_rcnn_0.5: 0.000000 2024-04-06 17:32:58,430 INFO recall_roi_0.7: 0.000000 2024-04-06 17:32:58,430 INFO recall_rcnn_0.7: 0.000000 2024-04-06 17:32:58,441 INFO Average predicted number of objects(7518 samples): 8.798 2024-04-06 17:32:58,779 INFO None 2024-04-06 17:32:58,779 INFO Result is saved to /hdd_10tb/sina/Radial_MAE/output/kitti_models/pv_rcnn/default/eval/epoch_78/test/default 2024-04-06 17:32:58,779 INFO ****Evaluation done.*****

kitti_dataset.yaml:

DATASET: 'KittiDataset' DATA_PATH: '../data/kitti'

POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1]

DATA_SPLIT: { 'train': train, 'test': test }

INFO_PATH: { 'train': [kitti_infos_train.pkl], 'test': [kitti_infos_test.pkl], }

GET_ITEM_LIST: ["points"] FOV_POINTS_ONLY: True

DATA_AUGMENTOR: DISABLE_AUG_LIST: ['placeholder'] AUG_CONFIG_LIST:

NAME: gt_sampling USE_ROAD_PLANE: True DB_INFO_PATH:
- kitti_dbinfos_train.pkl PREPARE: { filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'], filter_by_difficulty: [-1], }
  
  SAMPLE_GROUPS: ['Car:20','Pedestrian:15', 'Cyclist:15'] NUM_POINT_FEATURES: 4 DATABASE_WITH_FAKELIDAR: False REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0] LIMIT_WHOLE_SCENE: True
NAME: random_world_flip ALONG_AXIS_LIST: ['x']
NAME: random_world_rotation WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]
NAME: random_world_scaling WORLD_SCALE_RANGE: [0.95, 1.05]

POINT_FEATURE_ENCODING: { encoding_type: absolute_coordinates_encoding, used_feature_list: ['x', 'y', 'z', 'intensity'], src_feature_list: ['x', 'y', 'z', 'intensity'], }

DATA_PROCESSOR:

NAME: mask_points_and_boxes_outside_range REMOVE_OUTSIDE_BOXES: True
NAME: shuffle_points SHUFFLE_ENABLED: { 'train': True, 'test': False }
NAME: transform_points_to_voxels VOXEL_SIZE: [0.05, 0.05, 0.1] MAX_POINTS_PER_VOXEL: 5 MAX_NUMBER_OF_VOXELS: { 'train': 16000, 'test': 40000 }

pv_rcnn.yaml:

CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']

DATA_CONFIG: _BASECONFIG: cfgs/dataset_configs/kitti_dataset.yaml DATA_AUGMENTOR: DISABLE_AUG_LIST: ['placeholder'] AUG_CONFIG_LIST:

NAME: gt_sampling USE_ROAD_PLANE: False DB_INFO_PATH:

kitti_dbinfos_train.pkl PREPARE: { filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'], filter_by_difficulty: [-1], }

  SAMPLE_GROUPS: ['Car:15','Pedestrian:10', 'Cyclist:10']
  NUM_POINT_FEATURES: 4
  DATABASE_WITH_FAKELIDAR: False
  REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
  LIMIT_WHOLE_SCENE: False

NAME: random_world_flip ALONG_AXIS_LIST: ['x']
NAME: random_world_rotation WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]
NAME: random_world_scaling WORLD_SCALE_RANGE: [0.95, 1.05]

MODEL: NAME: PVRCNN

VFE:
    NAME: MeanVFE

BACKBONE_3D:
    NAME: VoxelBackBone8x

MAP_TO_BEV:
    NAME: HeightCompression
    NUM_BEV_FEATURES: 256

BACKBONE_2D:
    NAME: BaseBEVBackbone

    LAYER_NUMS: [5, 5]
    LAYER_STRIDES: [1, 2]
    NUM_FILTERS: [128, 256]
    UPSAMPLE_STRIDES: [1, 2]
    NUM_UPSAMPLE_FILTERS: [256, 256]

DENSE_HEAD:
    NAME: AnchorHeadSingle
    CLASS_AGNOSTIC: False

    USE_DIRECTION_CLASSIFIER: True
    DIR_OFFSET: 0.78539
    DIR_LIMIT_OFFSET: 0.0
    NUM_DIR_BINS: 2

    ANCHOR_GENERATOR_CONFIG: [
        {
            'class_name': 'Car',
            'anchor_sizes': [[3.9, 1.6, 1.56]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-1.78],
            'align_center': False,
            'feature_map_stride': 8,
            'matched_threshold': 0.6,
            'unmatched_threshold': 0.45
        },
        {
            'class_name': 'Pedestrian',
            'anchor_sizes': [[0.8, 0.6, 1.73]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-0.6],
            'align_center': False,
            'feature_map_stride': 8,
            'matched_threshold': 0.5,
            'unmatched_threshold': 0.35
        },
        {
            'class_name': 'Cyclist',
            'anchor_sizes': [[1.76, 0.6, 1.73]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-0.6],
            'align_center': False,
            'feature_map_stride': 8,
            'matched_threshold': 0.5,
            'unmatched_threshold': 0.35
        }
    ]

    TARGET_ASSIGNER_CONFIG:
        NAME: AxisAlignedTargetAssigner
        POS_FRACTION: -1.0
        SAMPLE_SIZE: 512
        NORM_BY_NUM_EXAMPLES: False
        MATCH_HEIGHT: False
        BOX_CODER: ResidualCoder

    LOSS_CONFIG:
        LOSS_WEIGHTS: {
            'cls_weight': 1.0,
            'loc_weight': 2.0,
            'dir_weight': 0.2,
            'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
        }

PFE:
    NAME: VoxelSetAbstraction
    POINT_SOURCE: raw_points
    NUM_KEYPOINTS: 2048
    NUM_OUTPUT_FEATURES: 128
    SAMPLE_METHOD: FPS

    FEATURES_SOURCE: ['bev', 'x_conv1', 'x_conv2', 'x_conv3', 'x_conv4', 'raw_points']
    SA_LAYER:
        raw_points:
            MLPS: [[16, 16], [16, 16]]
            POOL_RADIUS: [0.4, 0.8]
            NSAMPLE: [16, 16]
        x_conv1:
            DOWNSAMPLE_FACTOR: 1
            MLPS: [[16, 16], [16, 16]]
            POOL_RADIUS: [0.4, 0.8]
            NSAMPLE: [16, 16]
        x_conv2:
            DOWNSAMPLE_FACTOR: 2
            MLPS: [[32, 32], [32, 32]]
            POOL_RADIUS: [0.8, 1.2]
            NSAMPLE: [16, 32]
        x_conv3:
            DOWNSAMPLE_FACTOR: 4
            MLPS: [[64, 64], [64, 64]]
            POOL_RADIUS: [1.2, 2.4]
            NSAMPLE: [16, 32]
        x_conv4:
            DOWNSAMPLE_FACTOR: 8
            MLPS: [[64, 64], [64, 64]]
            POOL_RADIUS: [2.4, 4.8]
            NSAMPLE: [16, 32]

POINT_HEAD:
    NAME: PointHeadSimple
    CLS_FC: [256, 256]
    CLASS_AGNOSTIC: True
    USE_POINT_FEATURES_BEFORE_FUSION: True
    TARGET_CONFIG:
        GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
    LOSS_CONFIG:
        LOSS_REG: smooth-l1
        LOSS_WEIGHTS: {
            'point_cls_weight': 1.0,
        }

ROI_HEAD:
    NAME: PVRCNNHead
    CLASS_AGNOSTIC: True

    SHARED_FC: [256, 256]
    CLS_FC: [256, 256]
    REG_FC: [256, 256]
    DP_RATIO: 0.3

    NMS_CONFIG:
        TRAIN:
            NMS_TYPE: nms_gpu
            MULTI_CLASSES_NMS: False
            NMS_PRE_MAXSIZE: 9000
            NMS_POST_MAXSIZE: 512
            NMS_THRESH: 0.8
        TEST:
            NMS_TYPE: nms_gpu
            MULTI_CLASSES_NMS: False
            NMS_PRE_MAXSIZE: 1024
            NMS_POST_MAXSIZE: 100
            NMS_THRESH: 0.7

    ROI_GRID_POOL:
        GRID_SIZE: 6
        MLPS: [[64, 64], [64, 64]]
        POOL_RADIUS: [0.8, 1.6]
        NSAMPLE: [16, 16]
        POOL_METHOD: max_pool

    TARGET_CONFIG:
        BOX_CODER: ResidualCoder
        ROI_PER_IMAGE: 128
        FG_RATIO: 0.5

        SAMPLE_ROI_BY_EACH_CLASS: True
        CLS_SCORE_TYPE: roi_iou

        CLS_FG_THRESH: 0.75
        CLS_BG_THRESH: 0.25
        CLS_BG_THRESH_LO: 0.1
        HARD_BG_RATIO: 0.8

        REG_FG_THRESH: 0.55

    LOSS_CONFIG:
        CLS_LOSS: BinaryCrossEntropy
        REG_LOSS: smooth-l1
        CORNER_LOSS_REGULARIZATION: True
        LOSS_WEIGHTS: {
            'rcnn_cls_weight': 1.0,
            'rcnn_reg_weight': 1.0,
            'rcnn_corner_weight': 1.0,
            'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
        }

POST_PROCESSING:
    RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
    SCORE_THRESH: 0.1
    OUTPUT_RAW_SCORE: False

    EVAL_METRIC: kitti

    NMS_CONFIG:
        MULTI_CLASSES_NMS: False
        NMS_TYPE: nms_gpu
        NMS_THRESH: 0.1
        NMS_PRE_MAXSIZE: 4096
        NMS_POST_MAXSIZE: 500

OPTIMIZATION: BATCH_SIZE_PER_GPU: 2 NUM_EPOCHS: 80

OPTIMIZER: adam_onecycle
LR: 0.01
WEIGHT_DECAY: 0.01
MOMENTUM: 0.9

MOMS: [0.95, 0.85]
PCT_START: 0.4
DIV_FACTOR: 10
DECAY_STEP_LIST: [35, 45]
LR_DECAY: 0.1
LR_CLIP: 0.0000001

LR_WARMUP: False
WARMUP_EPOCH: 1

GRAD_NORM_CLIP: 10

Environment: Package Version Editable project location

absl-py 0.15.0 addict 2.4.0 aiofiles 22.1.0 aiosqlite 0.19.0 anyio 3.7.1 appdirs 1.4.4 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.2.3 astunparse 1.6.3 attrs 23.2.0 Babel 2.14.0 backcall 0.2.0 backports.functools-lru-cache 2.0.0 beautifulsoup4 4.12.3 bleach 6.0.0 Bottleneck 1.3.5 cached-property 1.5.2 cachetools 5.3.2 ccimport 0.4.2 certifi 2024.2.2 cffi 1.15.1 charset-normalizer 3.3.2 click 8.1.7 colorama 0.4.6 comm 0.1.4 cumm-cu113 0.4.11 cycler 0.11.0 debugpy 1.6.3 decorator 5.1.1 defusedxml 0.7.1 deprecation 2.1.0 descartes 1.1.0 docker-pycreds 0.4.0 easydict 1.11 entrypoints 0.4 exceptiongroup 1.2.0 faiss-cpu 1.7.4 fastjsonschema 2.19.1 filelock 3.12.2 fire 0.5.0 flatbuffers 1.12 fonttools 4.38.0 fqdn 1.5.1 fvcore 0.1.5.post20210915 gast 0.4.0 gdown 4.4.0 gitdb 4.0.11 GitPython 3.1.41 google-auth 2.27.0 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 grpcio 1.34.1 h5py 3.7.0 idna 3.6 imageio 2.31.2 importlib-metadata 6.7.0 importlib-resources 5.12.0 iopath 0.1.9 ipykernel 6.16.2 ipython 7.33.0 ipython-genutils 0.2.0 ipywidgets 8.1.2 isoduration 20.11.0 jedi 0.19.1 Jinja2 3.1.3 joblib 1.3.2 json5 0.9.14 jsonpointer 2.4 jsonschema 4.17.3 jupyter 1.0.0 jupyter_client 7.4.9 jupyter-console 6.6.3 jupyter_core 4.12.0 jupyter-events 0.5.0 jupyter_packaging 0.12.3 jupyter-server 1.24.0 jupyter_server_fileid 0.9.1 jupyter_server_ydoc 0.8.0 jupyter-ydoc 0.2.5 jupyterlab 3.6.7 jupyterlab-pygments 0.2.2 jupyterlab_server 2.24.0 jupyterlab_widgets 3.0.10 keras-nightly 2.5.0.dev2021032900 Keras-Preprocessing 1.1.2 kiwisolver 1.4.5 lark 1.1.9 llvmlite 0.39.1 Markdown 3.4.4 MarkupSafe 2.1.5 matplotlib 3.5.2 matplotlib-inline 0.1.3 mistune 3.0.2 motmetrics 1.4.0 nbclassic 1.0.0 nbclient 0.7.0 nbconvert 7.6.0 nbformat 5.8.0 nest_asyncio 1.6.0 networkx 2.6.3 ninja 1.11.1.1 notebook 6.5.6 notebook_shim 0.2.3 numba 0.56.4 numexpr 2.7.3 numpy 1.19.5 nuscenes-devkit 1.0.5 oauthlib 3.2.2 open3d 0.15.2 opencv-python 4.6.0.66 opt-einsum 3.3.0 packaging 23.2 pandas 1.3.5 pandocfilters 1.5.1 parso 0.8.3 patsy 0.5.6 pccm 0.4.11 pcdet 0.6.0+255db8f /hdd_10tb/sina/Radial_MAE pexpect 4.9.0 pickleshare 0.7.5 Pillow 9.2.0 pip 24.0 pkgutil_resolve_name 1.3.10 plotly 5.8.1 plotly-express 0.4.1 portalocker 2.6.0 prometheus-client 0.17.1 prompt-toolkit 3.0.42 protobuf 3.20.1 psutil 5.9.3 ptyprocess 0.7.0 pyasn1 0.5.1 pyasn1-modules 0.3.0 pybind11 2.11.1 pycparser 2.21 Pygments 2.17.2 pyparsing 3.1.1 pyquaternion 0.9.9 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 python-json-logger 2.0.7 pytorch3d 0.6.2 pytz 2024.1 PyWavelets 1.3.0 PyYAML 6.0 pyzmq 24.0.1 qtconsole 5.4.4 QtPy 2.4.1 requests 2.31.0 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rsa 4.9 scikit-image 0.19.3 scikit-learn 1.0.2 scipy 1.5.3 Send2Trash 1.8.2 sentry-sdk 1.40.3 setproctitle 1.3.3 setuptools 62.3.3 shapely 2.0.3 SharedArray 3.1.0 six 1.15.0 smmap 5.0.1 sniffio 1.3.0 soupsieve 2.4.1 spconv-cu113 2.3.6 statsmodels 0.13.2 tabulate 0.9.0 tenacity 8.2.3 tensorboard 2.11.2 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorboardX 2.5.1 tensorflow 2.5.0 tensorflow-estimator 2.5.0 termcolor 1.1.0 terminado 0.17.1 threadpoolctl 3.1.0 tifffile 2021.11.2 timm 0.4.5 tinycss2 1.1.1 tomli 2.0.1 tomlkit 0.11.0 torch 1.10.2 torch-scatter 2.1.1 torchvision 0.11.3 tornado 6.2 tqdm 4.64.0 traitlets 5.9.0 transforms3d 0.3.1 typing_extensions 4.7.1 uri-template 1.3.0 urllib3 2.0.7 wandb 0.16.3 waymo-open-dataset-tf-2-5-0 1.4.1 wcwidth 0.2.10 webcolors 1.13 webencodings 0.5.1 websocket-client 1.6.1 Werkzeug 2.2.3 wheel 0.42.0 widgetsnbextension 4.0.10 wrapt 1.12.1 xmltodict 0.13.0 y-py 0.6.2 yacs 0.1.8 ypy-websocket 0.8.4 zipp 3.15.0

Dai-Yu-Chen commented 5 months ago

Hi, I have the same problem. Have you solved this problem?

sinatayebati commented 5 months ago

Hi, I have the same problem. Have you solved this problem?

Unfortunately not!

github-actions[bot] commented 4 months ago

This issue is stale because it has been open for 30 days with no activity.

calcualatexzy commented 4 months ago

It is very annoying that I saw the same problem in "issues" several times and the team keeps on closing the issues due to 30 days stale, so is there any solution?

josyulavt commented 4 months ago

I have the same problem during training the loss is shown to be 0.2, I wanted to see the test results and everything is 0.

Any help is much appreciated

github-actions[bot] commented 3 months ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 2 months ago

This issue was closed because it has been inactive for 14 days since being marked as stale.

open-mmlab / OpenPCDet

Issue with KITTI "test set" evaluation turning all 0 values #1596