github-actions[bot]
commented
5 months ago 
Walkthrough
The changes primarily focus on enhancing the data pipeline architecture, introducing new schemas for devices, automated table creation, ingestion of stream tables, syncing colony information, block analyses, and SciViz updates. Significant modifications have been made to existing lookup tables like Lab, Location, User, and Arena. The codebase has been refactored for better readability and maintainability. New functionalities related to automated ingestion, worker configurations, and data stream handling have been introduced. Additionally, there are updates in the function add_arena_setup() for experiment 0.1.
Changes
| Files | Summary |
|---|---|
| .pre-commit-config.yaml | Updated the regex pattern for files and adjusted indentation in the Pyright configuration section. |
| aeon/dj_pipeline/* | Enhanced data pipeline architecture, core tables, and operating procedures. Introduced new schemas for devices, automated table creation, ingestion of stream tables, syncing colony information, block analyses, and SciViz updates. |
| aeon/dj_pipeline/create_experiments/* | Ingestion of metadata updated, specifically related to cameras, food patches, and weight scales. Function add_arena_setup() for experiment 0.1 updated. |
| aeon/dj_pipeline/ingest/process.py, aeon/dj_pipeline/populate/process.py, aeon/dj_pipeline/populate/worker.py | Removed script related to the Aeon project's DataJoint pipeline ingestion process. Introduced a script for auto-processing routines to operate the DataJoint pipeline for the Aeon project. |
| aeon/dj_pipeline/lab.py | Introduced new schemas, tables, and data structures related to lab information, user details, arena shapes, and equipment. Modified existing lookup tables like Lab, Location, User, and Arena. |
| aeon/dj_pipeline/scripts/* | Refactored code for cloning and archiving schemas and data for experiment 0.1, including changes in variable assignments and formatting. |
| aeon/dj_pipeline/utils/paths.py | Added from __future__ import annotations. Updated function signatures to include type hints and return type. Improved error message formatting. |
| aeon/dj_pipeline/utils/video.py | Introduced a new function retrieve_video_frames that retrieves video frames from a specified raw data directory, encodes them as JPEG images, and returns the frames along with metadata. |
| aeon/io/api.py | Modified the reindexing process by removing the method parameter and adding a dropna operation to handle missing values. |
| aeon/io/device.py | Refactored the compositeStream function to use a registry variable instead of composite. Renamed Device class attributes for clarity. |
| aeon/io/reader.py | Added a new Pose class for reading Harp-binarized tracking data, along with related methods for processing the data. |
| aeon/schema/schemas.py | Added new device schemas for different experiments (exp01, exp02), octagon experiment (octagon01), and social experiments (social01, social02). |
| aeon/schema/social.py | Removed a Pose class and related functions for reading tracking data. Modified the get_config_file function to accept an optional list of config file names. |
| aeon/schema/social_01.py | Added a new class RfidEvents that extends Stream with a constructor to initialize the stream path. |
| aeon/schema/social_02.py | Introduced new classes for handling different types of data streams within the Environment, SubjectData, Pose, WeightRaw, WeightFiltered, Patch, and RfidEvents groups. |
| aeon/util.py | Added from __future__ import annotations at the beginning of the file. |
| docker/cron_script.bash | Provided a bash script for automated ingestion and management of workers using Docker-compose. |
| aeon/analysis/block_plotting.py | Added functions for generating color gradients based on input values. Defined color schemes and markers for visualization purposes. |
Major updates
social0.1andsocial0.2based on the new Device/Stream object modelSummary by CodeRabbit