python bag api writer: time rewrite handling does not work with MCAP plugin

[fnobis]

I use the rosbag API to re-order timestamps in a ros2 bag mcap file. In the original bag, the messages are not ordered according to their timestamps. I use the header timestamp to rewrite the writer timestamp and thereby reorder the messages in the bag using the APIs mentioned here: https://mcap.dev/docs/python/ros2_example.html

This works as intended when I use sqlite3 as the output format. Messages receive a new order. When I use mcap as the output format, the messages keep their original order and are not sorted by the new timestamp. My workaround for now is to first convert the bag to sqlite3 with new timestamps and then convert again to mcap without changing the timestamps.

[jtbandes]

Could you please share the code you're using to do the conversion?

[jtbandes]

The timestamp is piped into the mcap file here: https://github.com/ros-tooling/rosbag2_storage_mcap/blob/b7b37693d7f423c9c1508fc869432f74fb1c273d/rosbag2_storage_mcap/src/mcap_storage.cpp#L519

[fnobis]

I added a gist of the script https://gist.github.com/fnobis/03790b7b6f69b63a2b04d7b13a4fab4a

In the example, the header timestamp is not changed. The header timestamps of both the camera and the camera_markers topics are the same. I would like to order them in the bag so that the camera_markers message is always logged to disk 1ms later than the image. I set the new "bag timestamp" (not sure if that is the logTime?) by adding a Duration to the original timestamp manually in the writer, this works with the sqlite3 option. With mcap the ordering is not changed.

[jtbandes]

And how do you know the order is not changed in the new bag? How are you testing the order?

[fnobis]

For testing, I open the converted bag again with the ros bag reader api and print the topic names for each message that is read. For the mcap bag, I get the same output as the original bag:

camera
camera_markers
camera_markers 
camera

for the sqlite3 bag the output is: as intended

camera
camera_markers
camera
camera_markers

[amacneil]

rosbag2_storage_mcap always plays back messages in the order they are written in the file, whereas the sqlite plugin will play back in timestamp order. We are planning to fix this, but there was no ticket so I created one here.

For now you can work around this by writing the messages to the file in the order you would like them to be played back.

FYI @emersonknapp

[fnobis]

Thanks for clarifying and opening that ticket!

[amacneil]

@fnobis fyi @james-rms implemented https://github.com/ros-tooling/rosbag2_storage_mcap/issues/50 today, so in the next release of rosbag2_storage_mcap (or if you build from source) you should be able to play back in log_time order.

As I mentioned, in the meantime you can also work around this by changing the order which messages are written to the file.

I'm going to close this issue now, but please let us know if there is still an issue you are having.

[fnobis]

Thanks for the update, I tried to run the new version from source with ros galactic in an empty workspace. During colcon build, I get the following error:

Starting >>> rosbag2_storage_mcap
--- stderr: rosbag2_storage_mcap                          
CMake Error at CMakeLists.txt:83 (find_package):
  By not providing "Findrosbag2_test_common.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "rosbag2_test_common", but CMake did not find one.

  Could not find a package configuration file provided by
  "rosbag2_test_common" with any of the following names:

    rosbag2_test_commonConfig.cmake
    rosbag2_test_common-config.cmake

  Add the installation prefix of "rosbag2_test_common" to CMAKE_PREFIX_PATH
  or set "rosbag2_test_common_DIR" to a directory containing one of the above
  files.  If "rosbag2_test_common" provides a separate development package or
  SDK, be sure it has been installed.

---

I have the package ros-galactic-rosbag2 installed via apt.

[emersonknapp]

The test packages are not part of the rosbag2 metapackage - they are not intended to be part of runtime distributions. Manually installing dependencies is not generally recommended, you can use rosdep install --from-paths src/ --ignore-src -y for example as a way to install dependencies for your given source tree.

That said, the dependency you’re looking for is ros-DISTRO-rosbag2-test-common