Exporting datasets as Rosbags.

[tonioteran]

(I'll just throw this out there in case it might be useful)

Rosbags (https://wiki.ros.org/rosbag) are pretty much a standard in the robotics in terms of datasets. They are also a pretty useful playback tool and help out a bunch with algorithm development and stuff. It's also a convenient way to make sure we package all of our information in a consistent, timestamped, and calibrated fashion.

Not sure what tools we currently use to do these sorts of things (like data playback and stuff, I know @clydemcqueen already does things like these without any ROS interaction), but it might be interesting to consider this avenue (it think it would definitely help in making our datasets more appealing, at least within the robotics community).

[clydemcqueen]

I can think of 2 ways to build rosbags:

1. Generate them from existing logs (dataflash logs, telemetry logs, still images) after the dive.
2. Enable the experimental Micro ROS support in ArduPilot and build some system (e.g., a BlueOS extension) that will capture a rosbag during the dive.

I just looked at the state of Micro ROS in ArduPilot. It is pretty cool and has great promise, but it is early days:

• The motivation behind the recent work was a GSoC project focused on Rover. The list of published messages is limited. It includes IMU (accel and gyro) and some EKF outputs (pose, vel, and lat/lng) but is missing barometer inputs, DVL inputs and USBL inputs.
• The publishing rates are hardcoded and aren't tied to sensor rates. The IMU topic publisher will sleep for 5ms between writes, but the DDS thread handles all incoming and outgoing messages so the 200Hz rate is an upper bound. In my SITL test I only saw 120Hz for the IMU message.
• The timestamps might jump from microseconds-since-boot to GPS-time when the USBL kicks in, so we would need to fix this or post-process the bags to trim off the pre-USBL messages.
• Of course we will also need to add in the images post-dive.

All-in-all, if we decide to build rosbags, I think the better option is to build them from existing logs, at least for now.

[tonioteran]

All-in-all, if we decide to build rosbags, I think the better option is to build them from existing logs, at least for now.

yup, 100% agree with this, at least of the time being

we can revisit this issue once we have a couple of datasets pretty neatly standardized, calibrated, and properly timestamped

the actual code to craft the rosbag should be pretty trivial; i've done it a couple of time either in python or cpp, we can use whichever one we deem better

[clydemcqueen]

I have started to do the investigations and write some scripts here: https://github.com/clydemcqueen/dive_to_bag

This has 3 scripts:

• a script to pull accel, gyro, mag and baro (TODO) out of dataflash logs
• a script to pull gps and dvl out of telemetry logs
• a script to open an mp4 file and pull out images

I decided to get the gps and dvl messages from the telemetry logs, as that is the primary source and the one I am familiar with, but it is also available in the dataflash logs.

Some caveats:

• users need to figure out the time shift (--boot time) for dataflash logs so they don't start at 0 in the unix epoch
• same for mp4 file (--start time), though getting a good timesync is still an open topic
• currently I am seeing IMU (accel, gyro) at 25Hz. We need a PR to ArduSub to log IMU messages at the loop rate (200Hz)

But it is a start!