loss do not converge using GD-MAE for nuScenes dataset

I try to implement GD-MAE on nuScenes dataset but the loss could not converge. The minimum loss is around 25. My config is like follows. You may provide some instructions on possible errors in the config file of gd_mae.yaml in nuscenes_models. .

CLASS_NAMES: ['car','truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']

DATA_CONFIG: _BASECONFIG: cfgs/dataset_configs/nuscenes_dataset.yaml

POINT_CLOUD_RANGE: [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]
BACKEND: &BACKEND
  NAME: HardDiskBackend
DATA_AUGMENTOR:
    DISABLE_AUG_LIST: ['placeholder']
    AUG_CONFIG_LIST:
        - NAME: gt_sampling
          BACKEND:
              <<: *BACKEND
          DB_INFO_PATH:
              - nuscenes_dbinfos_10sweeps_withvelo.pkl
          PREPARE: {
            filter_by_min_points: [
                'car:5','truck:5', 'construction_vehicle:5', 'bus:5', 'trailer:5',
                'barrier:5', 'motorcycle:5', 'bicycle:5', 'pedestrian:5', 'traffic_cone:5'
            ],
          }

          SAMPLE_GROUPS: [
              'car:2','truck:3', 'construction_vehicle:7', 'bus:4', 'trailer:6',
              'barrier:2', 'motorcycle:6', 'bicycle:6', 'pedestrian:2', 'traffic_cone:2'
          ]

          NUM_POINT_FEATURES: 5
          DATABASE_WITH_FAKELIDAR: False
          REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
          LIMIT_WHOLE_SCENE: True

        - NAME: random_world_flip
          PROBABILITY: 0.5
          ALONG_AXIS_LIST: ['x', 'y']

        - NAME: random_world_rotation
          PROBABILITY: 1.0
          WORLD_ROT_ANGLE: [-0.3925, 0.3925]

        - NAME: random_world_scaling
          PROBABILITY: 1.0
          WORLD_SCALE_RANGE: [0.95, 1.05]

POINT_FEATURE_ENCODING: {
    encoding_type: absolute_coordinates_encoding,
    used_feature_list: ['x', 'y', 'z', 'intensity'],
    src_feature_list:  ['x', 'y', 'z', 'intensity'],
}

DATA_PROCESSOR:
    -   NAME: mask_points_and_boxes_outside_range
        REMOVE_OUTSIDE_BOXES: True

    -   NAME: shuffle_points
        SHUFFLE_ENABLED: {
            'train': True,
            'test': True
        }

    -   NAME: calculate_grid_size
        VOXEL_SIZE: [0.32, 0.32, 8.0]

MODEL: NAME: CenterPoint

VFE:
    NAME: DynVFE
    TYPE: mean
    WITH_DISTANCE: False
    USE_ABSLOTE_XYZ: True
    USE_CLUSTER_XYZ: True
    MLPS: [[64, 128]]

BACKBONE_3D:
    NAME: SPTBackbone

    SST_BLOCK_LIST:
      - NAME: sst_block_x1
        PREPROCESS:
          WINDOW_SHAPE: [8, 8, 1]
          DROP_INFO: {
            'train': {
              '0': {'max_tokens': 16, 'drop_range': [0, 16]},
              '1': {'max_tokens': 32, 'drop_range': [16, 32]},
              '2': {'max_tokens': 64, 'drop_range': [32, 100000]}
            },
            'test': {
              '0': {'max_tokens': 16, 'drop_range': [0, 16]},
              '1': {'max_tokens': 32, 'drop_range': [16, 32]},
              '2': {'max_tokens': 64, 'drop_range': [32, 100000]}
            }
          }
          SHUFFLE_VOXELS: False
          POS_TEMPERATURE: 1000
          NORMALIZE_POS: False

        ENCODER:
          NUM_BLOCKS: 2
          STRIDE: 1
          D_MODEL: 128
          NHEAD: 8
          DIM_FEEDFORWARD: 256
          DROPOUT: 0.0
          ACTIVATION: "gelu"
          LAYER_CFG: {
            'cosine': True,
            'tau_min': 0.01
          }

      - NAME: sst_block_x2
        PREPROCESS:
          WINDOW_SHAPE: [8, 8, 1]
          DROP_INFO: {
            'train': {
              '0': {'max_tokens': 16, 'drop_range': [0, 16]},
              '1': {'max_tokens': 32, 'drop_range': [16, 32]},
              '2': {'max_tokens': 64, 'drop_range': [32, 100000]}
            },
            'test': {
              '0': {'max_tokens': 16, 'drop_range': [0, 16]},
              '1': {'max_tokens': 32, 'drop_range': [16, 32]},
              '2': {'max_tokens': 64, 'drop_range': [32, 100000]}
            }
          }
          SHUFFLE_VOXELS: False
          POS_TEMPERATURE: 1000
          NORMALIZE_POS: False

        ENCODER:
          NUM_BLOCKS: 2
          STRIDE: 2
          D_MODEL: 256
          NHEAD: 8
          DIM_FEEDFORWARD: 512
          DROPOUT: 0.0
          ACTIVATION: "gelu"
          LAYER_CFG: {
            'cosine': True,
            'tau_min': 0.01
          }

      - NAME: sst_block_x4
        PREPROCESS:
          WINDOW_SHAPE: [8, 8, 1]
          DROP_INFO: {
            'train': {
              '0': {'max_tokens': 16, 'drop_range': [0, 16]},
              '1': {'max_tokens': 32, 'drop_range': [16, 32]},
              '2': {'max_tokens': 64, 'drop_range': [32, 100000]}
            },
            'test': {
              '0': {'max_tokens': 16, 'drop_range': [0, 16]},
              '1': {'max_tokens': 32, 'drop_range': [16, 32]},
              '2': {'max_tokens': 64, 'drop_range': [32, 100000]}
            }
          }
          SHUFFLE_VOXELS: False
          POS_TEMPERATURE: 1000
          NORMALIZE_POS: False

        ENCODER:
          NUM_BLOCKS: 2
          STRIDE: 2
          D_MODEL: 256
          NHEAD: 8
          DIM_FEEDFORWARD: 512
          DROPOUT: 0.0
          ACTIVATION: "gelu"
          LAYER_CFG: {
            'cosine': True,
            'tau_min': 0.01
          }

    FEATURES_SOURCE: ['x_conv1', 'x_conv2', 'x_conv3']
    FUSE_LAYER:
      x_conv1:
        UPSAMPLE_STRIDE: 1
        NUM_FILTER: 128
        NUM_UPSAMPLE_FILTER: 128
      x_conv2:
        UPSAMPLE_STRIDE: 2
        NUM_FILTER: 256
        NUM_UPSAMPLE_FILTER: 128
      x_conv3:
        UPSAMPLE_STRIDE: 4
        NUM_FILTER: 256
        NUM_UPSAMPLE_FILTER: 128

BACKBONE_2D:
    NAME: SSTBEVBackbone
    NUM_FILTER: 128
    CONV_KWARGS: [
      {'out_channels': 128, 'kernel_size': 3, 'dilation': 1, 'padding': 1, 'stride': 1},
      {'out_channels': 128, 'kernel_size': 3, 'dilation': 1, 'padding': 1, 'stride': 1},
      {'out_channels': 128, 'kernel_size': 3, 'dilation': 2, 'padding': 2, 'stride': 1},
      {'out_channels': 128, 'kernel_size': 3, 'dilation': 1, 'padding': 1, 'stride': 1},
    ]
    CONV_SHORTCUT: [0, 1, 2]

DENSE_HEAD:
    NAME: CenterHead
    CLASS_AGNOSTIC: False

    CLASS_NAMES_EACH_HEAD: [
        ['car'], 
        ['truck', 'construction_vehicle'],
        ['bus', 'trailer'],
        ['barrier'],
        ['motorcycle', 'bicycle'],
        ['pedestrian', 'traffic_cone'],
    ]

    SHARED_CONV_CHANNEL: 64
    USE_BIAS_BEFORE_NORM: True
    NUM_HM_CONV: 2
    SEPARATE_HEAD_CFG:
        HEAD_ORDER: ['center', 'center_z', 'dim', 'rot', 'vel']
        HEAD_DICT: {
            'center': {'out_channels': 2, 'num_conv': 2},
            'center_z': {'out_channels': 1, 'num_conv': 2},
            'dim': {'out_channels': 3, 'num_conv': 2},
            'rot': {'out_channels': 2, 'num_conv': 2},
            'vel': {'out_channels': 2, 'num_conv': 2},
        }

    TARGET_ASSIGNER_CONFIG:
        FEATURE_MAP_STRIDE: 4
        NUM_MAX_OBJS: 500
        GAUSSIAN_OVERLAP: 0.1
        MIN_RADIUS: 2

    LOSS_CONFIG:
        LOSS_WEIGHTS: {
            'cls_weight': 1.0,
            'loc_weight': 0.25,
            'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.2, 0.2, 1.0, 1.0]
        }

    POST_PROCESSING:
        SCORE_THRESH: 0.1
        POST_CENTER_LIMIT_RANGE: [-61.2, -61.2, -10.0, 61.2, 61.2, 10.0]
        MAX_OBJ_PER_SAMPLE: 500
        NMS_CONFIG:
            NMS_TYPE: nms_gpu
            NMS_THRESH: 0.2
            NMS_PRE_MAXSIZE: 1000
            NMS_POST_MAXSIZE: 83

POST_PROCESSING:
    RECALL_THRESH_LIST: [0.3, 0.5, 0.7]

    EVAL_METRIC: kitti

OPTIMIZATION: BATCH_SIZE_PER_GPU: 4 NUM_EPOCHS: 30

OPTIMIZER: adam_onecycle
LR: 0.003
WEIGHT_DECAY: 0.01
MOMENTUM: 0.9

MOMS: [0.95, 0.85]
PCT_START: 0.4
DIV_FACTOR: 10
DECAY_STEP_LIST: [35, 45]
LR_DECAY: 0.1
LR_CLIP: 0.0000001

LR_WARMUP: False
WARMUP_EPOCH: 1

GRAD_NORM_CLIP: 10

Nightmare-n / GD-MAE

loss do not converge using GD-MAE for nuScenes dataset #9