Apollo 5.0: How to extract PCD files from a cyber record file?

[javangent]

We appreciate you go through Apollo documentations and search previous issues before creating an new one. If neither of the sources helped you with your issues, please report the issue using the following form. Please note missing info can delay the response time.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 14.04): Ubuntu 16.04
• Apollo installed from (source or binary): docker
• Apollo version (1.0, 1.5, 2.0, 2.5, 3.0): 5.0

Supporting materials (screenshots, command lines, code/script snippets):

I'm not quite sure how to extract binary PCD files from cyber record files with a lidar channel such as /apollo/sensor/lidar128/compensator/PointCloud2.

When searching the codebase, I found modules/localization/msf/local_tool/data_extraction/cyber_record_parser but this seems to give me the same error for any input I give it:

Another tool I found modules/perception/lidar/tools/exporter/msg_exporter requires some kind of config file of which I'm unsure of.

Any suggestions would be appreciated.

[javangent]

I just revisited the problem and noticed that /apollo/bazel-bin/modules/localization/msf/local_tool/data_extraction/cyber_record_parser actually reads the arguments correctly, when all necessary arguments have been specified.

I now ran into my next problem as when I run the cyber_record_parser with my cyber record in the following way:

/apollo/bazel-bin/modules/localization/msf/local_tool/data_extraction/cyber_record_parser \
--bag_file /apollo/data/bag/newrecording/new_recording.record.00000 \
--out_folder /apollo/data/bag/output/ \
--cloud_topic /apollo/sensor/lidar128/compensator/PointCloud2

I get the following error message:

WARNING: Logging before InitGoogleLogging() is written to STDERR
I1115 12:20:36.407627 24877 pcd_exporter.cc:44] []Compensated pcd callback.
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/repeated_field.h:1482] CHECK failed: (index) < (current_size_): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) < (current_size_): 
Aborted (core dumped)

Also, this is what cyber_recorder info outputs on my recording:

record_file:    ./data/bag/newrecording/new_recording.record.00000
version:        1.0
duration:       60.106406 Seconds
begin_time:     2019-10-11 17:28:15
end_time:       2019-10-11 17:29:15
size:           106389436 Bytes (101.460876 MB)
is_complete:    true
message_number: 602
channel_number: 1
channel_info:   
                /apollo/sensor/lidar128/compensator/PointCloud2        602 messages: apollo.drivers.PointCloud

Here is the backtrace on the core using GDB:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/apollo/bazel-bin/modules/localization/msf/local_tool/data_extraction/cyber_rec'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f14d345fc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007f14d345fc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f14d3463028 in __GI_abort () at abort.c:89
#2  0x00007f14d3a68535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f14d3a666d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f14d3a66703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f14d3a66922 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f14dda087f3 in google::protobuf::internal::LogMessage::Finish() ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libexternal_Scom_Ugoogle_Uprotobuf_Slibprotobuf_Ulite.so
#7  0x00007f14dda0882e in google::protobuf::internal::LogFinisher::operator=(google::protobuf::internal::LogMessage&) ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libexternal_Scom_Ugoogle_Uprotobuf_Slibprotobuf_Ulite.so
#8  0x00007f14ef323ef5 in google::protobuf::RepeatedPtrField<apollo::drivers::PointXYZIT>::TypeHandler::Type const& google::protobuf::internal::RepeatedPtrFieldBase::Get<google::protobuf::RepeatedPtrField<apollo::drivers::PointXYZIT>::TypeHandler>(int) const ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libmodules_Slocalization_Smsf_Slocal_Utool_Sdata_Uextraction_Slibdata_Uextraction.so
#9  0x00007f14ef322ff6 in google::protobuf::RepeatedPtrField<apollo::drivers::PointXYZIT>::Get(int) const ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libmodules_Slocalization_Smsf_Slocal_Utool_Sdata_Uextraction_Slibdata_Uextraction.so
#10 0x00007f14ef321e14 in apollo::drivers::PointCloud::point(int) const ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libmodules_Slocalization_Smsf_Slocal_Utool_Sdata_Uextraction_Slibdata_Uextraction.so
#11 0x00007f14ef3217f2 in apollo::localization::msf::PCDExporter::WritePcdFile(std::string const&, apollo::drivers::PointCloud const&) ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libmodules_Slocalization_Smsf_Slocal_Utool_Sdata_Uextraction_Slibdata_Uextraction.so
#12 0x00007f14ef321566 in apollo::localization::msf::PCDExporter::CompensatedPcdCallback(std::string const&) ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libmodules_Slocalization_Smsf_Slocal_Utool_Sdata_Uextraction_Slibdata_Uextraction.so
#13 0x000000000040f73a in main::{lambda(std::string const&)#1}::operator()(std::string const&) const ()
#14 0x0000000000410dfd in std::_Function_handler<void (std::string const&), main::{lambda(std::string const&)#1}>::_M_invoke(std::_Any_data const&, std::string const&) ()
#15 0x00007f14ef31cd27 in std::function<void (std::string const&)>::operator()(std::string const&) const ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libmodules_Slocalization_Smsf_Slocal_Utool_Sdata_Uextraction_Slibdata_Uextraction.so
#16 0x00007f14ef31b5a1 in apollo::localization::msf::CyberRecordReader::Read(std::string const&) ()
   from /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/local-fastbuild/bin/modules/localization/msf/local_tool/data_extraction/../../../../../_solib_k8libmodules_Slocalization_Smsf_Slocal_Utool_Sdata_Uextraction_Slibdata_Uextraction.so
#17 0x00000000004104a4 in main ()

[zhouyao4321]

The function WritePcdFile uses the width/height or point size of the input message to calculate the point size of output cloud. Please help to check that they are set accurately in the input message.

[javangent]

After encountering the error a year ago, I found this issue was caused by LGSVL publishing pointcloud messages with width: 11520 and height:1, which was more than the actual number of points in the message (7670).

A workaround for this problem was also provided here: https://github.com/ApolloAuto/apollo/issues/11936

[GabrielZZZ]

Hello @javangent What you have provided here is inspiring. I am also working on extracting PCD files from a record file, but I am now using Apollo 6.0 (master branch). And I noticed that some files are not in the same location you mentioned here when you used Apollo 5.0.

Can you take a look at it and give me some guidance on how to extract under Apollo 6.0?

Many thanks!

P.S. Here is the screenshot of my console when I followed your step above:

[javangent]

Hi @GabrielZZZ It seems your command contains backslashes ( \ ) even though your command is entirely on a single line (no newlines). Try removing the backslashes, it should fix the issue.

You could also alternatively change \ --bag_file to \--bag_file, but the first option would make more sense.