search

ros / ros_comm

ROS communications-related packages, including core client libraries (roscpp, rospy, roslisp) and graph introspection tools (rostopic, rosnode, rosservice, rosparam).
http://wiki.ros.org/ros_comm
747 stars 914 forks source link

Possibility to limit rosbag open memory requirements #2323

Closed kis-balazs closed 1 year ago

kis-balazs commented 1 year ago

Hi!

We are looking into extracting .csv files from ROS bags, and some of the bags are quite large (>40Gb). We tried limiting the soft limit of the memory usage for the python process executing - not to exceed certain required thresholds. This triggers a very specific MemoryException piping into "rosbag.bag.ROSBagUnindexedException: Unindexed bag" exception.

Do you have any recommendations how to limit/handle this in a better way? :)

Thank you!

kis-balazs commented 1 year ago

Hi, any update on this? :)

mikepurvis commented 1 year ago

@kis-balazs Questions of a "how do I" nature are better suited for answers.ros.org— this is a bug/feature tracker.

That said, I would try to implement your csv-extractor in C++ instead, as I believe the C++ interface to rosbag is more disciplined about not trying to load the whole thing into memory: http://wiki.ros.org/rosbag/Code%20API#C.2B-.2B-_API