Open boyang9602 opened 4 months ago
I filtered the logs of map node creation and found that they are more than the map nodes saved on my disk. Is it normal? I think some failed to be saved.
I0708 12:07:00.701506 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388)
I0708 12:07:00.701527 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388)
I0708 12:07:38.299526 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42389)
I0708 12:07:41.942131 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389)
I0708 12:08:19.930117 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388)
I0708 12:08:52.351574 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389)
I0708 12:09:55.961023 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42390)
I0708 12:10:07.810717 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390)
I0708 12:11:59.540479 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391)
I0708 12:11:59.541335 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391)
I0708 12:12:43.696748 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391)
I0708 12:12:52.404991 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390)
I0708 12:14:12.582288 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0708 12:18:31.129946 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387)
I0708 12:18:43.172639 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387)
I0708 12:19:48.452354 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387)
I0708 12:20:01.860620 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42386)
I0708 12:20:07.019407 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42386)
I0708 12:21:03.511037 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388)
I0708 12:22:18.799569 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42389)
I0708 12:24:17.612236 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390)
I0708 12:25:14.039001 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391)
I0708 12:25:57.306106 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390)
I0708 12:26:25.059980 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42391)
I0708 12:27:13.015880 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389)
I0708 12:33:14.966228 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42392)
I0708 12:33:41.273514 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42391)
I0708 12:34:01.273018 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42392)
I0708 12:34:04.753726 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42392)
I0708 12:34:54.257635 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
I0708 12:35:47.301105 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389)
I0708 12:36:40.341264 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42388)
I0708 12:37:27.624478 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42387)
I0708 12:37:28.094240 3025943 base_map.cc:245] []Created map node: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42387)
These are the files on my disk
13:33 $ ls -R
.:
00042386 00042387 00042388 00042389 00042390 00042391
./00042386:
00003559
./00042387:
00003558 00003559 00003560
./00042388:
00003557 00003558 00003559 00003560
./00042389:
00003556 00003557 00003558 00003559 00003560 00003561
./00042390:
00003556 00003557 00003558 00003559 00003560 00003561
./00042391:
00003556 00003557 00003558 00003559 00003560
So there are 34 creation logs but only 25 nodes saved on the disk.
Node (3556, 42390) is created twice based on the log (there is only one in the disk);
Node (3556, 42387), (3557, 42387), (3560, 42386), (3556, 42388), (3561, 42391), (3556, 42392), (3557, 42392), (3558, 42392) are missed from the disk.
Here are the logs for node save:
I0708 12:27:13.151298 3041418 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388).
I0708 12:29:43.605373 3041422 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391).
I0708 12:32:05.497555 3041427 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391).
I0708 12:32:43.976125 3041425 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390).
I0708 12:33:15.073017 3041423 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389).
I0708 12:33:41.390843 3041427 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388).
I0708 12:34:01.388948 3041429 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42388).
I0708 12:34:04.861517 3041430 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42387).
I0708 12:34:54.377503 3041421 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42387).
I0708 12:35:47.400295 3041430 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42387).
I0708 12:35:48.958354 3041423 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42386).
I0708 12:36:40.453127 3041433 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42389).
I0708 12:37:27.762301 3041420 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42389).
I0708 12:37:28.226308 3041432 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42390).
I0708 12:37:38.797415 3041425 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42390).
I0708 12:38:41.125078 3041421 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42391).
I0708 12:38:41.269021 3041433 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42390).
I0708 12:38:41.462898 3041428 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42389).
I0708 12:38:41.599717 3041420 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3559 Northing: 42391).
I0708 12:38:41.777155 3041422 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42391).
I0708 12:38:42.024045 3041424 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390).
I0708 12:38:42.163154 3041421 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42389).
I0708 12:38:42.313695 3041428 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42389).
I0708 12:38:42.471117 3041427 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42388).
I0708 12:38:42.682554 3041423 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42388).
I0708 12:38:42.849473 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42389).
I0708 12:38:42.963425 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42391).
I0708 12:38:43.074884 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42390).
I0708 12:38:43.179292 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42391).
I0708 12:38:43.304237 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42390).
I0708 12:38:43.417482 3025943 base_map_node.cc:66] []Save Map Node to disk: Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42388).
There are 31 logs for node save, while 6 are for the same node, (3557, 42388), (3558, 42388), (3557, 42389), (3557, 42389), (3557, 42391). So the map nodes on the disk all correspond to the logs.
Is it normal that some map nodes are created but not saved on the disk?
Since the ischanged is false by fault, I guess it is normal? https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_node.cc#L33-L54
I tracked the change of the map nodes For
Node (3556, 42390) is created twice based on the log (there is only one in the disk); Node (3556, 42387), (3557, 42387), (3560, 42386), (3556, 42388), (3561, 42391), (3556, 42392), (3557, 42392), (3558, 42392) are missed from the disk.
Node (3556, 42390) is created twice
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42390)
16:40:13.960451: created
16:58:42.222790: finalized:unchanged
17:00:44.082115: created
17:04:29.792712: finalized:changed
17:04:29.901326: saved
17:04:30.084803: loaded
17:04:31.216082: finalized:unchanged
Node (3556, 42387), (3557, 42387), (3560, 42386), (3556, 42388), (3561, 42391), (3556, 42392), (3557, 42392), (3558, 42392)
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42387)
17:03:17.186141: created
17:04:30.752338: finalized:unchanged
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42387)
17:03:16.703853: created
17:04:30.632504: finalized:unchanged
17:04:30.633667: finalized:unchanged
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3560 Northing: 42386)
16:46:06.322050: created
17:02:23.772083: finalized:unchanged
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42388)
17:02:29.378625: created
17:04:30.407481: finalized:unchanged
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3561 Northing: 42391)
16:52:22.415984: created
17:04:29.191915: finalized:unchanged
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3556 Northing: 42392)
16:59:51.409325: created
17:04:29.707600: finalized:unchanged
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3557 Northing: 42392)
16:59:05.801402: created
17:04:29.525470: finalized:unchanged
Map node (Resolution ID: 0 Zone ID: 32 Easting: 3558 Northing: 42392)
16:59:54.852206: created
17:04:29.751302: finalized:unchanged
We can see that (3557, 42387) has been created/loaded once but finalized twice. One Finalize
is called in the LoadMapNodeThreadSafety
of base_map.cc
https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map.cc#L228-L261
and the other is called in the Release
of base_map_pool.cc
https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_pool.cc#L45-L60
I thought it was the problem however it is not. LoadMapNodeThreadSafety
finalized it and reset it (to free the node in the pool), then Release
finalized it and deleted it.
More information:
Describe the bug Apollo v7 (9e1edfc) Failed to create the lidar localization map.
https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/scripts/msf_local_map_creator.sh#L12-L18
I think this error happened during/after the memory release stage.
Below is where the log
Save Map Node to disk: Map node ...
from https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_node.cc#L63-L68Below is where the log
Release free/busy node: Map node ...
from. (I added the logs at my local. ) https://github.com/ApolloAuto/apollo/blob/6edbb1f8fcee5ad2e75109b89de799236971548e/modules/localization/msf/local_pyramid_map/base_map/base_map_pool.cc#L45-L60After the Segmentation fault, the msf_local_map_creator continues to execute, but it hangs at the release stage and cannot exit normally after a while (it hangs after the logs below). Same or similar issue as https://github.com/ApolloAuto/apollo/issues/14897
To Reproduce I used KITTI odometry 00 sequence (raw_data 2011_10_03_drive_0027 synced)
bash scripts/msf_local_map_creator.sh KITTI/2011_10_03_drive_0027_sync/pcds/ KITTI/2011_10_03_drive_0027_sync/poses.txt 32 KITTI/2011_10_03_drive_0027_sync/
Screenshots
Desktop (please complete the following information):
Additional context This problem is similar to https://github.com/ApolloAuto/apollo/issues/11483 and https://github.com/ApolloAuto/apollo/issues/13214, however, I do not have
Scripts to convert KITTI point cloud to Apollo pcl XYZIT format
The first parameter is the input folder. It should be the path to point clouds of KITTI.
The second parameter is the output folder. It will be created if not exist.
It depends on https://github.com/daohu527/record_msg
Scripts to extract poses from oxts It depends on https://github.com/ApolloAuto/apollo/tree/master/modules/tools/adataset
The parameter
dataroot
is the path to2011_10_03_drive_0027_extract
.