ApolloAuto / apollo

An open autonomous driving platform
Apache License 2.0
25.25k stars 9.72k forks source link

cyber_recorder drops frames every 6 seconds #14613

Closed felixf4xu closed 2 years ago

felixf4xu commented 2 years ago

Describe the bug I'm recording data (6 cameras, 130k bytes each, 1 Lidar, 80 beams, all running at 10 fps) using cyber recorder. When the generated record file is parsed offline by python, some of the data frames are missing and it follows a similar pattern: for every 10-12 seconds, the data frames are dropped for about 6 seconds, for both cameras and lidar.

It can also be observed during the cyber_recorder record command, the status message will print how many messages are recorded and the numbers should increase over time, but it's easy to see the status message doen not update for a short time in every 10-12 seconds

But if observed by cyber_monitor, the data frame fps is always about 10fps

I'd like to seek some advice to check if 1) it's my data source issue, or 2) cyber::Write() issue, of 3) cyber recorder issue

Thanks.

daohu527 commented 2 years ago

@felixf4xu We need to provide detailed information to confirm the cause, at least so far I haven't found a similar problem.

felixf4xu commented 2 years ago

Our observation is that if cyber_recorder is writing .record file directly into a USB connected disk (like USB-SSD), it fails often.

Frankly speaking, it's understandable, but error or warning messages should be nice if messages are skipped to write.