search

SainsburyWellcomeCentre / aeon_mecha

Project Aeon's main library for interfacing with acquired data. Contains modules for raw data file io, data querying, data processing, data qc, database ingestion, and building computational data pipelines.
BSD 3-Clause "New" or "Revised" License
3 stars 5 forks source link

Create a "processed SLEAP data" table in database #332

Open jkbhagatio opened 7 months ago

jkbhagatio commented 7 months ago

Cleans up raw SLEAP assignments, e.g. by fixing impossible assignments.

ttngu207 commented 6 months ago

Proposed table design

@schema
class SubjectPosition(dj.Computed):
    definition = """  # 
    -> acquisition.Chunk
    """

    class Subject(dj.Part):
        definition = """
        -> master
        subject_name:          varchar(16)
        ---
        sample_count: int      # 
        x:          longblob   #     
        y:          longblob
        likelihood: longblob
        timestamps: longblob
        """

    class Part(dj.Part):
        definition = """
        -> master.Subject
        part_name: varchar(16)
        ---
        x:          longblob
        y:          longblob
        likelihood: longblob
        """

    def make(self, key):
        # SLEAP.PoseIdentity - subjects detected by SLEAP (for multiple cameras)
        # SubjectVisits (for "environment") - ground truth for subjects in the arena
        # RFID

        pass
ttngu207 commented 2 months ago

An update to this issue. We have discussed to apply the fix and store the results not only in the DataJoint pipeline, but instead to write the corrected SLEAP data into harp file with the same name as the original file, but placed in the "processed" folder instead.

This correction will be done on a per chunk basis - triggered using DataJoint as the workflow management tool, but again writing the results as files instead of storing in database.

Subsequently, DataJoint will load/ingest this corrected SLEAP data (in tracking.SLEAPTracking table).

Tagging @jkbhagatio , @lochhh

ttngu207 commented 2 months ago

First prototype in this branch: https://github.com/ttngu207/aeon_mecha

This prototype utilizes SLEAP identity correction method implemented by Chang Huan see here.

Currently, this method only works for 2 animals, we discussed potentially replacing this with the correction approach (using Kalman filter) developed first by Joaquin

See this table

jkbhagatio commented 2 months ago

This is almost done, lastly I just need to add a way to write out the pandas dataframe to the new binary output format of Bonsai.Sleap0.3