SainsburyWellcomeCentre / aeon_mecha

Project Aeon's main library for interfacing with acquired data. Contains modules for raw data file io, data querying, data processing, data qc, database ingestion, and building computational data pipelines.
BSD 3-Clause "New" or "Revised" License
6 stars 6 forks source link

Avoid expensive processing operations when loading dense pose data #424

Open glopesdev opened 1 month ago

glopesdev commented 1 month ago

While working on #421 I noticed we are currently doing a lot of manipulation and reorganization of the data at the level of the pose reader, specifically running apply over the entire data frame to collapse columns into a single dictionary:

https://github.com/SainsburyWellcomeCentre/aeon_mecha/blob/7812b4fbbda6fd679fae26c926be2f09baaae781/aeon/io/reader.py#L364-L374

This will unnecessarily slow down the parsing of the raw data (both the use of apply and allocation of many dictionaries), especially over long time intervals. Is this really necessary? I think the philosophy for readers should be as much as possible to simply load the raw data as-is. This code also scales poorly as performance will degrade the faster we run our cameras and the more possible identities we have.

We could add a flag similar to downsample for encoder data if we really wanted to preserve backwards-compatibility, but I feel that here we should rather do this dictionary transformation post-hoc in a utility function.

By doing this post-hoc we get the added benefit that we only run the coalescing code over the final "cropped" data in case we are reading in a time range.

glopesdev commented 1 month ago

Some relevant discussions: