search

SainsburyWellcomeCentre / aeon_mecha

Project Aeon's main library for interfacing with acquired data. Contains modules for raw data file io, data querying, data processing, data qc, database ingestion, and building computational data pipelines.
BSD 3-Clause "New" or "Revised" License
6 stars 6 forks source link

Refactor 'aeon' modules #237

Open jkbhagatio opened 1 year ago

jkbhagatio commented 1 year ago

In particular, './aeon/analysis/' should be removed, and modules within should be moved into './aeon/processing/' and the 'aeon_analysis' repo as appropriate (one argument could be that all plotting should belong in 'aeon_analysis' in which case in particular './aeon/analysis/plotting.py' should be moved to this repo).

e.g. 'aeon/processing/init.py' :

# This is where functions for common preprocessing done on raw data files will live
# (e.g. 'calc_forage_dist', 'calc_forage_bout', 'collate_movie', 'calc_tube_test', etc.)

Related to https://github.com/SainsburyWellcomeCentre/aeon_analysis/discussions/8

jkbhagatio commented 1 year ago

Actually, 'aeon/processing' could mirror 'aeon/schema', where we have a 'core' module, and then modules for different types of experiments (e.g. 'foraging', 'social', etc.)

And in this case, we could move 'aeon/schema/dataset.py' maybe to 'docs/examples/dataset.py' since 'dataset.py' does not define a collection of streams nor Reader classes, but instead contains examples of experiment datasets made up from streams.

jkbhagatio commented 1 year ago

We could even do the same with 'aeon/qc' - 'qc' functions dedicated to streams used in specific experiments.

Somehow I like the idea of this mirroring between 'schema', 'processing', and 'qc'

jkbhagatio commented 1 year ago

@glopesdev related to module refactoring - what is the differentiation between functions currently in 'aeon.io.video' and those in 'aeon.analysis.movies' ? Is there a reason you wanted to keep them separate rather than combining them into one module?

jkbhagatio commented 10 months ago

@glopesdev related to module refactoring - what is the differentiation between functions currently in 'aeon.io.video' and those in 'aeon.analysis.movies' ? Is there a reason you wanted to keep them separate rather than combining them into one module?

The reason for this is the functions in aeon.io.video particularly correspond to direct file io (read and write). The functions in aeon.io.video could potentially become methods in a Video reader.

jkbhagatio commented 10 months ago

Just brainstorming here, but now that we've agreed with steering committee that 'analysis' package can live in aeon_mecha, the repo structure could like something like this (note the core structure stays the same as it is currently):

| aeon/ |-- analysis/ |---- ... |-- dj_pipelines/ |---- ... |-- io/ |---- api.py |---- device.py |---- reader.py   # abstract classes only |-- processing/ |---- core.py |---- foraging.py |---- social.py |---- ... |-- qc/ |---- core.py |---- foraging.py |---- social.py |---- ... |-- schema/   # modules here correspond to general exp types |------ core.py |------ foraging.py |------ social.py |------ ...


Some notes:

  1. In aeon.schema, each module can contain Readers, binder functions, and helper functions, all with the goal of creating versioned schemas for the given experiment name. The __main__ in these modules can return different versions of a schema, depending on input args.

  2. aeon.io.schema.core can contain things that will get reused by all other schemas, including a potential raw Video reader.

  3. I would move the Pose reader and associated code from aeon.schema.social to aeon.schema.core