Need Help when reading the `.record` file

[lethal233]

Hi,

I met a problem when reading the record file.

OS runtime specification:

ubuntu 18.04
python 3.8

Here is the steps to reproduce:

1. Here, demo_3.5.record is the file downloaded from https://apollo-system.cdn.bcebos.com/dataset/6.0_edu/demo_3.5.record
2. write a test file:

   
   from cyber_record.record import Record

file_name = "./demo_3.5.record" record = Record(file_name) for topic, message, t in record.read_messages("/apollo/canbus/chassis_detail"): print("{}, {}, {}".format(topic, type(message), t))

3. output (no output):
```sh
[no output]

4. another test file:

   
   from cyber_record.record import Record

file_name = "./demo_3.5.record" record = Record(file_name) for topic, message, t in record.read_messages(): print("{}, {}, {}".format(topic, type(message), t))

5. also no output
6. using command line:
```bash
cyber_record info -f ./demo_3.5.record

7. output (right and appropriate):

   
   record_file: ./demo_3.5.record
   version:     1.0
   begin_time:  2019-01-07 11:12:57.338835
   end_time:    2019-01-07 11:13:42.886741
   duration:    45.55 s
   size:        91.79 MByte
   message_number: 61615
   channel_number: 32

/tf_static , apollo.transform.TransformStampeds , 0 /apollo/monitor , apollo.common.monitor.MonitorMessage , 9 /apollo/localization/pose , apollo.localization.LocalizationEstimate, 4558 /apollo/perception/obstacles , apollo.perception.PerceptionObstacles , 455 /apollo/planning , apollo.planning.ADCTrajectory , 455 /apollo/sensor/gnss/imu , apollo.drivers.gnss.Imu , 4553 /apollo/canbus/chassis_detail , apollo.canbus.ChassisDetail , 0 /apollo/hmi/status , apollo.dreamview.HMIStatus , 27 /apollo/routing_request , apollo.routing.RoutingRequest , 0 /apollo/sensor/gnss/rtk_eph , apollo.drivers.gnss.GnssEphemeris , 132 /tf , apollo.transform.TransformStampeds , 9079 /apollo/control , apollo.control.ControlCommand , 4539 /apollo/monitor/system_status , apollo.monitor.SystemStatus , 9 /apollo/prediction , apollo.prediction.PredictionObstacles , 453 /apollo/sensor/gnss/odometry , apollo.localization.Gps , 4537 /apollo/navigation , apollo.relative_map.NavigationInfo , 0 /apollo/localization/msf_gnss , apollo.localization.LocalizationEstimate, 45 /apollo/drive_event , apollo.common.DriveEvent , 0 /apollo/routing_response , apollo.routing.RoutingResponse , 0 /apollo/canbus/chassis , apollo.canbus.Chassis , 4536 /apollo/localization/msf_status , apollo.localization.LocalizationStatus, 4538 /apollo/sensor/gnss/best_pose , apollo.drivers.gnss.GnssBestPose , 45 /apollo/sensor/gnss/gnss_status , apollo.drivers.gnss.GnssStatus , 45 /apollo/sensor/gnss/corrected_imu , apollo.localization.CorrectedImu , 4536 /apollo/sensor/gnss/ins_stat , apollo.drivers.gnss.InsStat , 45 /apollo/sensor/gnss/rtk_obs , apollo.drivers.gnss.EpochObservation , 92 /apollo/sensor/gnss/raw_data , apollo.drivers.gnss.RawData , 13921 /apollo/perception/traffic_light , apollo.perception.TrafficLightDetection, 291 /apollo/localization/msf_lidar , apollo.localization.LocalizationEstimate, 180 /apollo/control/pad , apollo.control.PadMessage , 0 /apollo/guardian , apollo.guardian.GuardianCommand , 4535 /apollo/hmi/audio_capture , apollo.dreamview.AudioCapture , 0

8. using command line to read specific topic:
```bash
cyber_record echo -f ./demo_3.5.record -t /apollo/canbus/chassis_detail

9. no output either

I wonder if I am misunderstanding how to use your tool.

Please let me know if there is any additional information or resources that I can provide to assist with the investigation. I am willing to collaborate and support your team in any way possible.

[daohu527]

There is no problem in use. I tested and there is no output. I will check and feedback then

[lethal233]

There is no problem in use. I tested and there is no output. I will check and feedback then

Thank you!

[daohu527]

1. python3 test_read_record.py will work, why there's not message output is the message num is 0, you can use other messages and test again. You also can find the message num use cyber_record info -f xx.record
2. cyber_record echo -f not work because the record file is old version, and there's some info missing in it, I use other record and it work.

[lethal233]

• python3 test_read_record.py will work, why there's not message output is the message num is 0, you can use other messages and test again. You also can find the message num use cyber_record info -f xx.record
• cyber_record echo -f not work because the record file is old version, and there's some info missing in it, I use other record and it work.

Thanks for your quick response! I have some questions

• As you can see the output in step 6 & 7, the message num is 61615 instead of 0. And there is indeed an output. I do not know why there is inconsistency of output between command line cyber_record and python module (step 4 & 5).
• What is the oldest version that your tool possibly supports (or date)?

Best

[daohu527]

they use different API interface, which cyber_record echo -f will faster, and test_read_record.py will slow because it needs to traverse whole file, and the former does not, of course we can use tools to repair this record

[lethal233]

they use different API interface, which cyber_record echo -f will faster, and test_read_record.py will slow because it needs to traverse whole file, and the former does not, of course we can use tools to repair this record

Gotcha, thanks for your kindly help! If I have any question related to this issue, I will post it here. Let's close it.

[daohu527]

Glad your problem was solved! If you have any other questions about cyber_record, welcome to discuss : )

[boyang9602]

Hi @daohu527 ,

I recorded some data using Apollo v5 and wanted to parse the data but met the same problem. I wonder if there is any way to parse the old version data. For example, can it be solved by replacing the proto files with old versions?

[daohu527]

@boyang9602 Theoretically, the messages are compatible. However, due to the Python path issue, incompatibility may occur. I think it may be necessary to decide on a technical solution based on the specific reasons.

We'd better open a new question about this!

[boyang9602]

In case anyone meets the same issue, I found an alternative way for a temporary solution. Apollo has a python wrapper for cyber, which covers the cyber_recorder too. Below is a sample usage in Apollo v5 for LocalizationEstimate message

from cyber_py.record import RecordReader
from modules.localization.proto.localization_pb2 import LocalizationEstimate
reader = RecordReader('20240812162113.record.00000')
it = reader.read_messages()
msg = next(it)
# msg.topic, msg.message, msg.data_type, msg.timestamp
local_estimate = LocalizationEstimate()
local_estimate.ParseFromString(message.message)