search

usdot-fhwa-stol / carma-analytics-fotda

CARMA Analytics is a secure, scalable and cloud agnostic data architecture to enable research analysis on big data sets by defining repeatable and scalable processes for ingesting data, creation of a data repository to fuse, process and perform quality assurance.
Apache License 2.0
3 stars 3 forks source link

Update rosbag_processor.py #1

Open ianberg-volpe opened 3 years ago

ianberg-volpe commented 3 years ago

use built-in topic list function

PR Details

Description

hi @tyagi86, I was thinking about your comment in our meeting earlier this week about how the current processing script reads the entire bag into memory first, so I took a look at the script. I think this edit (based on the script Volpe has been using to get topic lists) would potentially fix that and hopefully make bag processing run a little faster / allow for larger bag files to be processed.

Related Issue

n/a

Motivation and Context

discussion in meeting on 2021-06-08 about preprocessing limitations for very large bagfiles

How Has This Been Tested?

I ran the lines of code I replaced as well as the replacement lines on the CARMA Analytics prod machine using the _2021-05-18-19-52-15_down-selected.bag rosbag as a test. The new code was much faster.

image

Types of changes

Checklist:

ianberg-volpe commented 3 years ago

I don't think we can get away with removing bag.read_messages() completely, the data has to get read out somewhere. But on line 50 it's only reading the topic you specify, which is much faster than reading the entire bagfile (as it was doing on line 36).