search

strands-project / mongodb_store

MongoDB tools for storing and analysing runs of ROS systems.
BSD 3-Clause "New" or "Revised" License
49 stars 72 forks source link

Support for capped collections #245

Closed gavanderhoorn closed 5 years ago

gavanderhoorn commented 5 years ago

This was mentioned in #8 as a possible approach to mimic "ringbuffers", but that issue is from 2014, so I figured I'd post a new one to see whether mongodb_store supports capped collections.

Use case would be a fixed-size "last N minutes" recording of all ROS traffic in an application for post-mortem diagnosis and debugging.

If this is currently not supported, could someone familiar with the internals of mongodb_store estimate what it would take to add support?

Edit: and related question: would replication be affected by using something like a capped collection?

hawesie commented 5 years ago

AFAIK we didn't end up implementating that for #8 (which I should probably close at this point), but since then they were added to mongodb: https://docs.mongodb.com/manual/core/capped-collections/

To use these you'd need to add a flag to mongodb_log or mongodb_store to either create or convert the collections used to a capped version. Perhaps capping requirements would be best done via rosparam reading in a yaml file of collection to cap mappings.

I don't think this would be especially hard to add this to the current setup given a reasonably ability to code in python.

gavanderhoorn commented 5 years ago

Ok, so there's nothing in the way mongodb_store or the other packages/nodes use the db that would prevent it from working with a capped collection that you know of?

hawesie commented 5 years ago

Nope. Under the hood mongodb_store just reads and writes to collections in the normal way.

gavanderhoorn commented 5 years ago

Great. Thanks for the insights.

If/when we decide to go this way, I'll try to update this issue with our experiences.