search

ApolloAuto / apollo

An open autonomous driving platform
Apache License 2.0
25.26k stars 9.72k forks source link

msf_local_map_creator: Segmentation fault (core dumped) #15460

Open boyang9602 opened 4 months ago

boyang9602 commented 4 months ago

Describe the bug Apollo v7 (9e1edfc) Failed to create the lidar localization map. 

scripts/msf_local_map_creator.sh: line 18: 2096581 Segmentation fault      (core dumped) $APOLLO_BIN_PREFIX/modules/localization/msf/local_tool/map_creation/lossless_map_creator --use_plane_inliers_only true --pcd_folders $1 --pose_files $2 --map_folder $4 --zone_id $3 --coordinate_type UTM --map_resolution_type single

https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/scripts/msf_local_map_creator.sh#L12-L18

I think this error happened during/after the memory release stage. 

I0707 16:17:02.475179 2148212 velodyne_utility.cc:47] []Un-organized-point-cloud
I0707 16:17:02.476176 2148212 lossless_map_creator.cc:273] []Loaded 1202463D Points at Trial: 0 Frame: 4543.
I0707 16:17:02.498037 2148212 extract_ground_plane.h:112] []the 0 interation: plane_num = 112
I0707 16:17:02.504361 2148212 extract_ground_plane.h:112] []the 1 interation: plane_num = 94
I0707 16:17:02.507581 2148212 extract_ground_plane.h:112] []the 2 interation: plane_num = 58
I0707 16:17:02.509807 2148212 extract_ground_plane.h:112] []the 3 interation: plane_num = 3
I0707 16:17:02.510290 2148212 extract_ground_plane.h:120] []plane_patch takes:0.026327sec.
I0707 16:17:02.510298 2148212 extract_ground_plane.h:122] []total_plane_num = 267
I0707 16:17:02.510300 2148212 extract_ground_plane.h:123] []total_points_num = 45393
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389)
I0707 16:17:02.535341 2148212 base_map_node.cc:284] []map node compress ratio: 0.0704835
I0707 16:17:02.558287 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390)
I0707 16:17:02.644779 2155545 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391).
I0707 16:17:02.666762 2148212 base_map_node.cc:284] []map node compress ratio: 0.0661397
I0707 16:17:02.689380 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387)
I0707 16:17:02.782181 2155545 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390).
I0707 16:17:02.803663 2148212 base_map_node.cc:284] []map node compress ratio: 0.0526841
I0707 16:17:02.826417 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387)
I0707 16:17:02.844969 2148212 base_map_node.cc:284] []map node compress ratio: 0.0277287
I0707 16:17:02.867584 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388)
I0707 16:17:02.954752 2155538 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389).
I0707 16:17:02.974113 2148212 base_map_node.cc:284] []map node compress ratio: 0.0378863
I0707 16:17:02.996620 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391)
I0707 16:17:03.093468 2155539 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391).
I0707 16:17:03.110529 2148212 base_map_node.cc:284] []map node compress ratio: 0.0167038
I0707 16:17:03.133288 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391)
I0707 16:17:03.149989 2148212 base_map_node.cc:284] []map node compress ratio: 0.0151782
I0707 16:17:03.172648 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390)
I0707 16:17:03.259934 2155542 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42391).
I0707 16:17:03.281219 2148212 base_map_node.cc:284] []map node compress ratio: 0.061117
I0707 16:17:03.303725 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390)
I0707 16:17:03.321890 2148212 base_map_node.cc:284] []map node compress ratio: 0.0284023
I0707 16:17:03.344622 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389)
I0707 16:17:03.366489 2148212 base_map_node.cc:284] []map node compress ratio: 0.0132794
I0707 16:17:03.389833 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42389)
I0707 16:17:03.485847 2155539 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390).
I0707 16:17:03.505115 2148212 base_map_node.cc:284] []map node compress ratio: 0.0350676
I0707 16:17:03.527673 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42389)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0707 16:17:03.621021 2155538 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389).
I0707 16:17:03.639901 2148212 base_map_node.cc:284] []map node compress ratio: 0.0344522
I0707 16:17:03.663535 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389)
I0707 16:17:03.775633 2155541 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389).
I0707 16:17:03.794324 2148212 base_map_node.cc:284] []map node compress ratio: 0.0265961
I0707 16:17:03.817782 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389)
I0707 16:17:03.929507 2155546 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388).
I0707 16:17:03.949863 2148212 base_map_node.cc:284] []map node compress ratio: 0.0463046
I0707 16:17:03.972491 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388)
I0707 16:17:03.994354 2148212 base_map_node.cc:284] []map node compress ratio: 0.0618036
I0707 16:17:04.017382 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388)
GetMapNodeSafe: This node don't exist in cache! 
load this node from disk now!Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388)
I0707 16:17:04.141695 2155532 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388).
I0707 16:17:04.163966 2148212 base_map_node.cc:284] []map node compress ratio: 0.0646691
I0707 16:17:04.186791 2148212 base_map.cc:247] []Loaded map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388)
I0707 16:17:04.187177 2148212 base_map_config.cc:48] []Saved the map configuration to: KITTI/2011_10_03_drive_0027_sync//lossless_map/config.xml
I0707 16:17:04.188860 2148212 base_map_pool.cc:52] []Release free node: Map node (Resolution ID: 374344960 Zone ID: 32516 Easting: 3557 Northing: 42387)
I0707 16:17:04.190112 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 374344928 Zone ID: 32516 Easting: 3558 Northing: 42388)
I0707 16:17:04.302307 2148212 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42389).
I0707 16:17:04.303580 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 374344928 Zone ID: 32516 Easting: 3558 Northing: 42389)
I0707 16:17:04.304772 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 374344928 Zone ID: 32516 Easting: 3556 Northing: 42387)
I0707 16:17:04.415647 2148212 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391).
I0707 16:17:04.416836 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 374345360 Zone ID: 32516 Easting: 3557 Northing: 42391)
I0707 16:17:04.523108 2148212 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42390).
I0707 16:17:04.524408 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 374344976 Zone ID: 32516 Easting: 3558 Northing: 42390)
I0707 16:17:04.525596 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391)
I0707 16:17:04.625166 2148212 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391).
I0707 16:17:04.626369 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391)
I0707 16:17:04.627557 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387)
I0707 16:17:04.628736 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390)
I0707 16:17:04.629936 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 374344928 Zone ID: 32516 Easting: 3561 Northing: 42389)
I0707 16:17:04.631119 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 374344928 Zone ID: 32516 Easting: 3561 Northing: 42390)
I0707 16:17:04.751824 2148212 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390).
I0707 16:17:04.753321 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390)
I0707 16:17:04.754871 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391)
I0707 16:17:04.756363 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42389)
I0707 16:17:04.757831 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0707 16:17:04.759249 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389)
I0707 16:17:04.760712 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389)
I0707 16:17:04.762189 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388)
I0707 16:17:04.859980 2148212 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388).
I0707 16:17:04.861238 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388)
I0707 16:17:04.862105 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389)
I0707 16:17:04.862982 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390)
I0707 16:17:04.863874 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387)
I0707 16:17:04.864755 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387)
I0707 16:17:04.865646 2148212 base_map_pool.cc:58] []Release busy node:Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388)
I0707 16:17:04.865655 2148212 base_map_pool.cc:62] []ALL released.
scripts/msf_local_map_creator.sh: line 18: 2096581 Segmentation fault      (core dumped) $APOLLO_BIN_PREFIX/modules/localization/msf/local_tool/map_creation/lossless_map_creator --use_plane_inliers_only true --pcd_folders $1 --pose_files $2 --map_folder $4 --zone_id $3 --coordinate_type UTM --map_resolution_type single

Below is where the log Save Map Node to disk: Map node ... from https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_node.cc#L63-L68

Below is where the log Release free/busy node: Map node ... from. (I added the logs at my local. ) https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_pool.cc#L45-L60

void BaseMapNodePool::Release() {
  if (node_reset_workers_.valid()) {
    node_reset_workers_.get();
  }
  for (BaseMapNode* node : free_list_) {
    FinalizeMapNode(node);
    DellocMapNode(node);
    AINFO << "Release free node: " << node->GetMapNodeIndex();
  }
  free_list_.clear();
  for (BaseMapNode* node : busy_nodes_) {
    FinalizeMapNode(node);
    DellocMapNode(node);
    AINFO << "Release busy node:" << node->GetMapNodeIndex();
  }
  busy_nodes_.clear();
  pool_size_ = 0;
  AINFO << "ALL released.";
}

After the Segmentation fault, the msf_local_map_creator continues to execute, but it hangs at the release stage and cannot exit normally after a while (it hangs after the logs below). Same or similar issue as https://github.com/ApolloAuto/apollo/issues/14897 

I0707 14:29:58.787117 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390)
I0707 14:29:58.872294 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390).
I0707 14:29:58.873261 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390)
I0707 14:29:58.943966 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390).
I0707 14:29:58.945029 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0707 14:29:59.014536 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387).
I0707 14:29:59.015617 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387)
I0707 14:29:59.084031 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387).
I0707 14:29:59.085242 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387)
I0707 14:29:59.164824 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387).
I0707 14:29:59.166018 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387)
I0707 14:29:59.249568 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388).
I0707 14:29:59.250713 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388)
I0707 14:29:59.333639 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388).
I0707 14:29:59.334698 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388)
I0707 14:29:59.417869 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388).
I0707 14:29:59.418757 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388)
I0707 14:29:59.494041 2109925 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388).
I0707 14:29:59.495286 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388)
I0707 14:29:59.496557 2109925 base_map_pool.cc:52] []Release free node: Map node (Resolution ID: 3209032384 Zone ID: 22076 Easting: 3560 Northing: 42389)
I0707 14:29:59.497820 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 3729505712 Zone ID: 32517 Easting: 3557 Northing: 42389)
I0707 14:29:59.499153 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 3729505504 Zone ID: 32517 Easting: 3558 Northing: 42389)
I0707 14:29:59.500486 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389)
I0707 14:29:59.501756 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389)
I0707 14:29:59.503041 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389)
I0707 14:29:59.504349 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391)
I0707 14:29:59.505858 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391)
I0707 14:29:59.507216 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391)
I0707 14:29:59.508461 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391)
I0707 14:29:59.509718 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42391)
I0707 14:29:59.510886 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42386)
I0707 14:29:59.512094 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390)
I0707 14:29:59.513289 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390)
I0707 14:29:59.514283 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42390)
I0707 14:29:59.515272 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390)
I0707 14:29:59.515978 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390)
I0707 14:29:59.516607 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0707 14:29:59.517236 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387)
I0707 14:29:59.517791 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387)
I0707 14:29:59.518352 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387)
I0707 14:29:59.518937 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388)
I0707 14:29:59.519495 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388)
I0707 14:29:59.520056 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388)
I0707 14:29:59.520689 2109925 base_map_pool.cc:58] []Release: busy nodeMap node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388)

To Reproduce I used KITTI odometry 00 sequence (raw_data 2011_10_03_drive_0027 synced)

  1. I converted the point cloud from KITTI to pcl format used in Apollo (XYZIT).
  2. I extracted poses from oxts following the required format 
    pose_file: a file that contains all poses of pcd files. Each line corresponds to a pcd file, and the format of each line is "pcd_number timestamp x y z qx qy qz qw". Thereinto, "pcd_number" corresponds to the pcd file name.
  3. I executed bash scripts/msf_local_map_creator.sh KITTI/2011_10_03_drive_0027_sync/pcds/ KITTI/2011_10_03_drive_0027_sync/poses.txt 32 KITTI/2011_10_03_drive_0027_sync/

Screenshots image

Desktop (please complete the following information):

Additional context This problem is similar to https://github.com/ApolloAuto/apollo/issues/11483 and https://github.com/ApolloAuto/apollo/issues/13214, however, I do not have

terminate called after throwing an instance of 'YAML::BadFile'
what(): bad file

Scripts to convert KITTI point cloud to Apollo pcl XYZIT format

The first parameter is the input folder. It should be the path to point clouds of KITTI.

The second parameter is the output folder. It will be created if not exist.

It depends on https://github.com/daohu527/record_msg

import sys
import pathlib
import numpy as np
from datetime import datetime

from record_msg import pypcd

def to_timestamp(sensor_time):
  date_sec, nano_sec = sensor_time.split('.')
  time_sec = datetime.strptime(date_sec, '%Y-%m-%d %H:%M:%S')
  return datetime.timestamp(time_sec) + float(nano_sec)*1e-9

def convert(filepath, outpath, timestamp, compression='binary_compressed'):
    # read
    if filepath.suffix == '.bin':
        scan = np.fromfile(filepath, dtype=np.float32)
    elif filepath.suffix == '.txt':
        scan = np.loadtxt(filepath)
    else:
        raise "Unsupported file extension. It has to be 'txt' or 'bin'"
    scan = scan.reshape(-1, 4)
    intensities = (scan[:, -1] * 255).astype(np.uint8)
    timestamps = np.ones_like(intensities, dtype=np.float64) * timestamp
    md = {'version': .7,
          'fields': ['x', 'y', 'z', 'intensity', 'timestamp'],
          'count': [1, 1, 1, 1, 1],
          'width': len(scan),
          'height': 1,
          'viewpoint': [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0],
          'points': len(scan),
          'size': [4, 4, 4, 1, 8],
          'type': ['F', 'F', 'F', 'U', 'F'],
          'data': compression}
    pc_data = np.hstack([scan[:, :3], intensities.reshape(-1, 1), timestamps.reshape(-1, 1)])
    pc = pypcd.PointCloud(md, pc_data)
    # save
    pc.save(outpath / f'{int(filepath.stem)}.pcd')

inpath = pathlib.Path(sys.argv[1])
outpath = pathlib.Path(sys.argv[2])
outpath.mkdir(parents=True, exist_ok=True)

timestamps = []
with (inpath.parent / 'timestamps.txt').open('r') as f:
    for line in f.readlines():
        timestamps.append(to_timestamp(line))

for filepath in inpath.iterdir():
    if filepath.is_file():
        convert(filepath, outpath, timestamps[int(filepath.stem)])

Scripts to extract poses from oxts It depends on https://github.com/ApolloAuto/apollo/tree/master/modules/tools/adataset

The parameter dataroot is the path to 2011_10_03_drive_0027_extract. 

from adataset.kitti.proj_helper import latlon2utm
from adataset.kitti.geometry import Euler
from adataset.kitti.kitti import KITTISchema
import sys

def convert(dataroot):
    kitti_schema = KITTISchema(dataroot)
    utm_zones = []
    poses = []
    for i, imu in enumerate(kitti_schema.imu_schemes()):
        utm_x, utm_y, utm_zone = latlon2utm(imu.lat, imu.lon)
        euler = Euler(imu.roll, imu.pitch, imu.yaw)
        q = euler.to_quaternion()
        utm_zones.append(utm_zone)
        # poses.append(f'{i:010d} {imu.timestamp} {utm_x} {utm_y} {imu.alt} {q.w} {q.x} {q.y} {q.z}\n')
        poses.append([i, imu.timestamp, utm_x, utm_y, imu.alt, q.w, q.x, q.y, q.z])
    return poses

if __name__ == '__main__':
    poses = convert(sys.argv[1])
    with open(sys.argv[1] + '/poses.txt', 'w') as f:
        for pose in poses:
            f.write(f'{pose[0]:010d} {pose[1]} {pose[2]} {pose[3]} {pose[4]} {pose[5]} {pose[6]} {pose[7]} {pose[8]}\n')
boyang9602 commented 4 months ago

I filtered the logs of map node creation and found that they are more than the map nodes saved on my disk. Is it normal? I think some failed to be saved.

I0708 12:07:00.701506 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388)
I0708 12:07:00.701527 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388)
I0708 12:07:38.299526 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42389)
I0708 12:07:41.942131 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389)
I0708 12:08:19.930117 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388)
I0708 12:08:52.351574 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389)
I0708 12:09:55.961023 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42390)
I0708 12:10:07.810717 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390)
I0708 12:11:59.540479 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391)
I0708 12:11:59.541335 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391)
I0708 12:12:43.696748 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391)
I0708 12:12:52.404991 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390)
I0708 12:14:12.582288 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0708 12:18:31.129946 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387)
I0708 12:18:43.172639 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387)
I0708 12:19:48.452354 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387)
I0708 12:20:01.860620 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42386)
I0708 12:20:07.019407 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42386)
I0708 12:21:03.511037 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388)
I0708 12:22:18.799569 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42389)
I0708 12:24:17.612236 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390)
I0708 12:25:14.039001 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391)
I0708 12:25:57.306106 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390)
I0708 12:26:25.059980 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42391)
I0708 12:27:13.015880 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389)
I0708 12:33:14.966228 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42392)
I0708 12:33:41.273514 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42391)
I0708 12:34:01.273018 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42392)
I0708 12:34:04.753726 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42392)
I0708 12:34:54.257635 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0708 12:35:47.301105 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389)
I0708 12:36:40.341264 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42388)
I0708 12:37:27.624478 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42387)
I0708 12:37:28.094240 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42387)

These are the files on my disk

13:33 $ ls -R
.:
00042386  00042387  00042388  00042389  00042390  00042391

./00042386:
00003559

./00042387:
00003558  00003559  00003560

./00042388:
00003557  00003558  00003559  00003560

./00042389:
00003556  00003557  00003558  00003559  00003560  00003561

./00042390:
00003556  00003557  00003558  00003559  00003560  00003561

./00042391:
00003556  00003557  00003558  00003559  00003560

So there are 34 creation logs but only 25 nodes saved on the disk.

Node (3556, 42390) is created twice based on the log (there is only one in the disk);
Node (3556, 42387), (3557, 42387), (3560, 42386), (3556, 42388), (3561, 42391), (3556, 42392), (3557, 42392), (3558, 42392) are missed from the disk.

Here are the logs for node save:

I0708 12:27:13.151298 3041418 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388).
I0708 12:29:43.605373 3041422 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391).
I0708 12:32:05.497555 3041427 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391).
I0708 12:32:43.976125 3041425 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390).
I0708 12:33:15.073017 3041423 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389).
I0708 12:33:41.390843 3041427 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388).
I0708 12:34:01.388948 3041429 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388).
I0708 12:34:04.861517 3041430 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387).
I0708 12:34:54.377503 3041421 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387).
I0708 12:35:47.400295 3041430 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387).
I0708 12:35:48.958354 3041423 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42386).
I0708 12:36:40.453127 3041433 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42389).
I0708 12:37:27.762301 3041420 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389).
I0708 12:37:28.226308 3041432 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390).
I0708 12:37:38.797415 3041425 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390).
I0708 12:38:41.125078 3041421 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391).
I0708 12:38:41.269021 3041433 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390).
I0708 12:38:41.462898 3041428 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389).
I0708 12:38:41.599717 3041420 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391).
I0708 12:38:41.777155 3041422 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42391).
I0708 12:38:42.024045 3041424 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390).
I0708 12:38:42.163154 3041421 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389).
I0708 12:38:42.313695 3041428 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389).
I0708 12:38:42.471117 3041427 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388).
I0708 12:38:42.682554 3041423 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388).
I0708 12:38:42.849473 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42389).
I0708 12:38:42.963425 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391).
I0708 12:38:43.074884 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42390).
I0708 12:38:43.179292 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391).
I0708 12:38:43.304237 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390).
I0708 12:38:43.417482 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388).

There are 31 logs for node save, while 6 are for the same node, (3557, 42388), (3558, 42388), (3557, 42389), (3557, 42389), (3557, 42391). So the map nodes on the disk all correspond to the logs.

Is it normal that some map nodes are created but not saved on the disk?

Since the ischanged is false by fault, I guess it is normal? https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_node.cc#L33-L54

https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_node.cc#L63-L68

boyang9602 commented 4 months ago

I tracked the change of the map nodes For

Node (3556, 42390) is created twice based on the log (there is only one in the disk); Node (3556, 42387), (3557, 42387), (3560, 42386), (3556, 42388), (3561, 42391), (3556, 42392), (3557, 42392), (3558, 42392) are missed from the disk.

Node (3556, 42390) is created twice

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
16:40:13.960451: created
16:58:42.222790: finalized:unchanged
17:00:44.082115: created
17:04:29.792712: finalized:changed
17:04:29.901326: saved
17:04:30.084803: loaded
17:04:31.216082: finalized:unchanged

Node (3556, 42387), (3557, 42387), (3560, 42386), (3556, 42388), (3561, 42391), (3556, 42392), (3557, 42392), (3558, 42392)

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42387)
17:03:17.186141: created
17:04:30.752338: finalized:unchanged

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42387)
17:03:16.703853: created
17:04:30.632504: finalized:unchanged
17:04:30.633667: finalized:unchanged

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42386)
16:46:06.322050: created
17:02:23.772083: finalized:unchanged

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42388)
17:02:29.378625: created
17:04:30.407481: finalized:unchanged

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42391)
16:52:22.415984: created
17:04:29.191915: finalized:unchanged

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42392)
16:59:51.409325: created
17:04:29.707600: finalized:unchanged

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42392)
16:59:05.801402: created
17:04:29.525470: finalized:unchanged

Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42392)
16:59:54.852206: created
17:04:29.751302: finalized:unchanged

We can see that (3557, 42387) has been created/loaded once but finalized twice. One Finalize is called in the LoadMapNodeThreadSafety of base_map.cc https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map.cc#L228-L261 and the other is called in the Release of base_map_pool.cc https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_pool.cc#L45-L60

I thought it was the problem however it is not. LoadMapNodeThreadSafety finalized it and reset it (to free the node in the pool), then Release finalized it and deleted it.

boyang9602 commented 3 months ago

More information:

  1. For different datasets, it has different situations: a. Sometimes it's core dumped b. sometimes it is hung and does not exit
  2. Apollo v5.0.0 works fine.