search

foxglove / mcap

MCAP is a modular, performant, and serialization-agnostic container file format, useful for pub/sub and robotics applications.
https://mcap.dev
MIT License
495 stars 90 forks source link

MCAP filter cli #351

Closed amacneil closed 2 years ago

amacneil commented 2 years ago

Ability to filter mcap files, similar to rosbag filter:

james-rms commented 2 years ago

What if instead we implemented a filter-friendly API in Python, and delivered a mcap_filter.py wrapper for simple CLI jobs?

Here's my thought process:

  1. robot data manipulation tasks exist on a smooth complexity continuum, from: a. simple filtering: select all messages by channel or time range b. more complex filtering: filter by a field in a message, like frame_id in a sensor msg or hardware_id in /diagnostics c. transformation: reassign all images to one topic, fix timestamps, etc. ideally users can use tools from the same ecosystem for all of the above.
  2. Python is a super popular language for data wrangling, in the ROS ecosystem especially but also more generally.
  3. Building a nice python API for mcap manipulation will probably be really nice for users.
  4. Once that's done, it's easy to implement a command-line filter tool in Python, which would also double as an example script for users who want to dip their toes into mcap manipulation in Python.
amacneil commented 2 years ago

I agree with the use cases you outlined. However, our CLI is very easy to install (brew install mcap), and at the very least there should be an easy way to split out a single topic or time range (since we already have the ability to merge files from the CLI - seems weird if I can't do the reverse).

Landing more advanced filter features in the python library might be a better place to do it (and improving our python library in general is good). But honestly I think we should ignore more advanced filtering for now until there are requests for it - plenty of other things to work on.

james-rms commented 2 years ago

This was completed in https://github.com/foxglove/mcap/pull/445