search

traveller59 / second.pytorch

SECOND for KITTI/NuScenes object detection
MIT License
1.72k stars 722 forks source link

TypeError: Object of type 'ndarray' is not JSON serializable #326

Open 451578928 opened 4 years ago

451578928 commented 4 years ago

I was use Titan V, and set batch size=5, num_workers=2.actully I tried batch_size =1 ,it wasn't work at all. Did anyone know how to solve this issue? my error as follow: `(second) cxt@ubuntu4-X299X-AORUS-MASTER:~/codetest/second.pytorch$ python second/pytorch/train.py train --config_path=second/configs/car.fhd.config --model_dir=second/model_saved /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning: Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.so.

For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup') warnings.warn(errors.NumbaWarning(msg)) /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning: Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice.

For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup') warnings.warn(errors.NumbaWarning(msg)) /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/cuda/envvars.py:17: NumbaWarning: Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_CUDA_DRIVER=/usr/lib/x86_64-linux-gnu/libcuda.so.

For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup') warnings.warn(errors.NumbaWarning(msg)) [ 41 1600 1408] num parameters: 69 False _amp_stash {'Car': 5} [-1] load 2207 Pedestrian database infos load 14357 Car database infos load 734 Cyclist database infos load 1297 Van database infos load 488 Truck database infos load 224 Tram database infos load 337 Misc database infos load 56 Person_sitting database infos After filter database: load 2207 Pedestrian database infos load 13442 Car database infos load 734 Cyclist database infos load 1297 Van database infos load 488 Truck database infos load 224 Tram database infos load 337 Misc database infos load 56 Person_sitting database infos feature_map_size [1, 200, 176] remain number of infos: 3712 feature_map_size [1, 200, 176] remain number of infos: 3769 model: { second: { network_class_name: "VoxelNet" voxel_generator { point_cloud_range : [0, -40, -3, 70.4, 40, 1]

point_cloud_range : [0, -32.0, -3, 52.8, 32.0, 1]

  voxel_size : [0.05, 0.05, 0.1]
  #voxel_size : [0.2, 0.2, 0.4]
  max_number_of_points_per_voxel : 5
  #max_number_of_points_per_voxel : 35
}

voxel_feature_extractor: {
  module_class_name: "SimpleVoxel"
  num_filters: [16]
  with_distance: false
  num_input_features: 4
}
middle_feature_extractor: {
  module_class_name: "SpMiddleFHD"
  # num_filters_down1: [] # protobuf don't support empty list.
  # num_filters_down2: []
  downsample_factor: 8
  num_input_features: 4
}
rpn: {
  module_class_name: "RPNV2"
  layer_nums: [5]
  layer_strides: [1]
  num_filters: [128]
  upsample_strides: [1]
  num_upsample_filters: [128]
  use_groupnorm: false
  num_groups: 32
  num_input_features: 128
}
loss: {
  classification_loss: {
    weighted_sigmoid_focal: {
      alpha: 0.25
      gamma: 2.0
      anchorwise_output: true
    }
  }
  localization_loss: {
    weighted_smooth_l1: {
      sigma: 3.0
      code_weight: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
    }
  }
  classification_weight: 1.0
  localization_weight: 2.0
}
num_point_features: 4 # model's num point feature should be independent of dataset
# Outputs
use_sigmoid_score: true
encode_background_as_zeros: true
encode_rad_error_by_sin: true
sin_error_factor: 1.0

use_direction_classifier: true # this can help for orientation benchmark
direction_loss_weight: 0.2 # enough.
num_direction_bins: 2
direction_limit_offset: 1

# Loss
pos_class_weight: 1.0
neg_class_weight: 1.0

loss_norm_type: NormByNumPositives
# Postprocess
post_center_limit_range: [0, -40, -2.2, 70.4, 40, 0.8]
nms_class_agnostic: false # only valid in multi-class nms

box_coder: {
  ground_box3d_coder: {
    linear_dim: false
    encode_angle_vector: false
  }
}
target_assigner: {
  class_settings: {
    anchor_generator_range: {
      sizes: [1.6, 3.9, 1.56] # wlh
      anchor_ranges: [0, -40.0, -1.00, 70.4, 40.0, -1.00] # carefully set z center
      rotations: [0, 1.57] # DON'T modify this unless you are very familiar with my code.
    }
    matched_threshold : 0.6
    unmatched_threshold : 0.45
    class_name: "Car"
    use_rotate_nms: true
    use_multi_class_nms: false
    nms_pre_max_size: 1000
    nms_post_max_size: 100
    nms_score_threshold: 0.3 # 0.4 in submit, but 0.3 can get better hard performance
    nms_iou_threshold: 0.01

    region_similarity_calculator: {
      nearest_iou_similarity: {
      }
    }
  }
  # anchor_generators: {
  #   anchor_generator_stride: {
  #     sizes: [1.6, 3.9, 1.56] # wlh
  #     strides: [0.4, 0.4, 0.0] # if generate only 1 z_center, z_stride will be ignored
  #     offsets: [0.2, -39.8, -1.00] # origin_offset + strides / 2
  #     rotations: [0, 1.57] # DON'T modify this unless you are very familiar with my code.
  #     matched_threshold : 0.6
  #     unmatched_threshold : 0.45
  #   }
  # }
  sample_positive_fraction : -1
  sample_size : 512
  assign_per_class: true
}

} }

train_input_reader: { dataset: { dataset_class_name: "KittiDataset" kitti_info_path: "/home/cxt/Kitti/object/kitti_infos_train.pkl" kitti_root_path: "/home/cxt/Kitti/object" }

batch_size: 6 preprocess: { max_number_of_voxels: 17000 shuffle_points: true num_workers: 2 groundtruth_localization_noise_std: [1.0, 1.0, 0.5]

groundtruth_rotation_uniform_noise: [-0.3141592654, 0.3141592654]

# groundtruth_rotation_uniform_noise: [-1.57, 1.57]
groundtruth_rotation_uniform_noise: [-0.78539816, 0.78539816]
global_rotation_uniform_noise: [-0.78539816, 0.78539816]
global_scaling_uniform_noise: [0.95, 1.05]
global_random_rotation_range_per_object: [0, 0] # pi/4 ~ 3pi/4
global_translate_noise_std: [0, 0, 0]
anchor_area_threshold: -1
remove_points_after_sample: true
groundtruth_points_drop_percentage: 0.0
groundtruth_drop_max_keep_points: 15
remove_unknown_examples: false
sample_importance: 1.0
random_flip_x: false
random_flip_y: true
remove_environment: false
database_sampler {
  database_info_path: "/home/cxt/Kitti/object/kitti_dbinfos_train.pkl"
  sample_groups {
    name_to_max_num {
      key: "Car"
      value: 15
    }
  }
  database_prep_steps {
    filter_by_min_num_points {
      min_num_point_pairs {
        key: "Car"
        value: 5
      }
    }
  }
  database_prep_steps {
    filter_by_difficulty {
      removed_difficulties: [-1]
    }
  }
  global_random_rotation_range_per_object: [0, 0]
  rate: 1.0
}

} }

train_config: { optimizer: { adam_optimizer: { learning_rate: { one_cycle: { lr_max: 2.25e-3 moms: [0.95, 0.85] div_factor: 10.0 pct_start: 0.4 } } weight_decay: 0.01 } fixed_weight_decay: true use_moving_average: false }

steps: 99040 # 1238 * 120

steps: 49520 # 619 * 80

steps: 30950 # 619 * 80

steps_per_eval: 3095 # 619 * 5

steps: 23200 # 464 50 steps_per_eval: 2320 # 619 5

save_checkpoints_secs : 1800 # half hour save_summary_steps : 10 enable_mixed_precision: false loss_scale_factor: -1 clear_metrics_every_epoch: true }

eval_input_reader: { dataset: { dataset_class_name: "KittiDataset" kitti_info_path: "/home/cxt/Kitti/object/kitti_infos_val.pkl"

kitti_info_path: "/media/yy/960evo/datasets/kitti/kitti_infos_test.pkl"

kitti_root_path: "/home/cxt/Kitti/object"

} batch_size: 6 preprocess: { max_number_of_voxels: 40000 shuffle_points: false num_workers: 2 anchor_area_threshold: -1 remove_environment: false } }

WORKER 0 seed: 1577546776 WORKER 1 seed: 1577546777 /home/cxt/codetest/second.pytorch/second/core/preprocess.py:463: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C)) points[i:i + 1, :3] = points[i:i + 1, :3] @ rot_mat_T[j] /home/cxt/codetest/second.pytorch/second/core/preprocess.py:463: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C)) points[i:i + 1, :3] = points[i:i + 1, :3] @ rot_mat_T[j] /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/typing/npydecl.py:977: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C)) warnings.warn(NumbaPerformanceWarning(msg)) /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/typing/npydecl.py:977: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C)) warnings.warn(NumbaPerformanceWarning(msg)) /home/cxt/codetest/second.pytorch/second/core/geometry.py:272: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Invalid use of Function() with argument(s) of type(s): (array(float32, 3d, C), (slice, list(int64), slice))

File "second/core/geometry.py", line 288: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    vec1 = polygon - polygon[:, [num_points_of_polygon - 1] +
                             list(range(num_points_of_polygon - 1)), :]
                             ^

@numba.jit /home/cxt/codetest/second.pytorch/second/core/geometry.py:272: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Invalid use of Function() with argument(s) of type(s): (array(float32, 3d, C), (slice, list(int64), slice))

File "second/core/geometry.py", line 288: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

    vec1 = polygon - polygon[:, [num_points_of_polygon - 1] +
                             list(range(num_points_of_polygon - 1)), :]
                             ^

@numba.jit /home/cxt/codetest/second.pytorch/second/core/geometry.py:272: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: cannot determine Numba type of <class 'numba.dispatcher.LiftedLoop'>

File "second/core/geometry.py", line 296: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

cross = 0.0
for i in range(num_points):
^

@numba.jit /home/cxt/codetest/second.pytorch/second/core/geometry.py:272: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: cannot determine Numba type of <class 'numba.dispatcher.LiftedLoop'>

File "second/core/geometry.py", line 296: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

cross = 0.0
for i in range(num_points):
^

@numba.jit /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/object_mode_passes.py:178: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

File "second/core/geometry.py", line 273: @numba.jit def points_in_convex_polygon_jit(points, polygon, clockwise=True): ^

state.func_ir.loc)) /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/object_mode_passes.py:178: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.

File "second/core/geometry.py", line 273: @numba.jit def points_in_convex_polygon_jit(points, polygon, clockwise=True): ^

state.func_ir.loc)) /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/object_mode_passes.py:187: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "second/core/geometry.py", line 273: @numba.jit def points_in_convex_polygon_jit(points, polygon, clockwise=True): ^

warnings.warn(errors.NumbaDeprecationWarning(msg, state.func_ir.loc)) /home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/numba/object_mode_passes.py:187: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "second/core/geometry.py", line 273: @numba.jit def points_in_convex_polygon_jit(points, polygon, clockwise=True): ^

warnings.warn(errors.NumbaDeprecationWarning(msg, state.func_ir.loc)) CUDA out of memory. Tried to allocate 104.00 MiB (GPU 0; 11.78 GiB total capacity; 1.06 GiB already allocated; 32.38 MiB free; 74.94 MiB cached) Traceback (most recent call last): File "second/pytorch/train.py", line 308, in train ret_dict = net_parallel(example_torch) File "/home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, *kwargs) File "/home/cxt/codetest/second.pytorch/second/pytorch/models/voxelnet.py", line 363, in forward preds_dict = self.network_forward(voxels, num_points, coors, batch_size_dev) File "/home/cxt/codetest/second.pytorch/second/pytorch/models/voxelnet.py", line 332, in network_forward voxel_features, coors, batch_size) File "/home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(input, *kwargs) File "/home/cxt/codetest/second.pytorch/second/pytorch/models/middle.py", line 206, in forward ret = ret.dense() File "/home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/spconv/init.py", line 85, in dense return res.permute(trans_params).contiguous() RuntimeError: CUDA out of memory. Tried to allocate 104.00 MiB (GPU 0; 11.78 GiB total capacity; 1.06 GiB already allocated; 32.38 MiB free; 74.94 MiB cached)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "second/pytorch/train.py", line 665, in fire.Fire() File "/home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/fire/core.py", line 138, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/home/cxt/anaconda3/envs/second/lib/python3.6/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace component = fn(*varargs, kwargs) File "second/pytorch/train.py", line 425, in train model_logging.log_text(json.dumps(example["metadata"], indent=2), step) File "/home/cxt/anaconda3/envs/second/lib/python3.6/json/init.py", line 238, in dumps kw).encode(obj) File "/home/cxt/anaconda3/envs/second/lib/python3.6/json/encoder.py", line 201, in encode chunks = list(chunks) File "/home/cxt/anaconda3/envs/second/lib/python3.6/json/encoder.py", line 428, in _iterencode yield from _iterencode_list(o, _current_indent_level) File "/home/cxt/anaconda3/envs/second/lib/python3.6/json/encoder.py", line 325, in _iterencode_list yield from chunks File "/home/cxt/anaconda3/envs/second/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict yield from chunks File "/home/cxt/anaconda3/envs/second/lib/python3.6/json/encoder.py", line 437, in _iterencode o = _default(o) File "/home/cxt/anaconda3/envs/second/lib/python3.6/json/encoder.py", line 180, in default o.class.name) TypeError: Object of type 'ndarray' is not JSON serializable `

simaker commented 4 years ago

RuntimeError: CUDA out of memory. Tried to allocate 104.00 MiB (GPU 0; 11.78 GiB total capacity; 1.06 GiB already allocated; 32.38 MiB free; 74.94 MiB cached)

you ran out of Memory