search

SainsburyWellcomeCentre / aeon_mecha

Project Aeon's main library for interfacing with acquired data. Contains modules for raw data file io, data querying, data processing, data qc, database ingestion, and building computational data pipelines.
BSD 3-Clause "New" or "Revised" License
6 stars 6 forks source link

New `ingestion_schemas` + MANY minor fixes and improvements #438

Closed ttngu207 closed 2 weeks ago

ttngu207 commented 1 month ago

This PR includes a few major new logic/features and many minor fixes (mostly for BlockAnalysis)

  1. Use ingestion_schemas - i.e. a separate set of schemas for DJ ingestion (specialized Encoder reader and Video reader)
    • Encoder reader with default downsampling to 50Hz
    • Video reader with only the hw_timestamp column
  2. Update streams.py DJ schema accordingly (mainly to drop extra columns when reading Video data)
  3. Add special handling to ingest fullpose data for social0.2
  4. Add a new tracking table: BlobPosition to read and store blob position tracking when SLEAP data is not available
  5. Fixes & improvements for BlockAnalysis
    • Bugfix incorrect extraction of subject_in_patch times
    • fetch_stream function rounds times to microseconds (mysql precision is to microseconds only)
    • BlockAnalysis use BlobPosition when SLEAPTracking is not available
    • improve logic to search for chunks in a given block
    • BlockDetection - when double 0s are found, use the first 0s (instead of the 2nd one)
    • Add BlockForaging computed table

Fix #427 Replace #437

ttngu207 commented 3 weeks ago

I've left a bunch of small comments here.

A general point - maybe add top-level module docstrings for all these modules? Unless this will be done by @MilagrosMarin in #443 ?

Yes, this will be address when we apply Ruff checks and fixes