Introduce explicit object model for device streams

As discussed in #334 this PR introduces a new API and object model for describing device stream schemas. Below we introduce the proposed API design, and provide some examples of usage. The proposed design is fully backwards compatible, so it can be used interchangeably with the previous approach of using functions for stream definition.

Detailed Design

The main change introduced by the new API is to replace functions and static classes with two base classes: Stream and StreamGroup.

Stream

The Stream class represents a single device stream, and defines the __iter__ method which is used to update the schema dictionary by automatically using the name of the class as dictionary key. This is the main advantage over the previous approach, where functions returned dictionaries with a single arbitrary key, which was both repetitive and error prone since there was no validation on the relationship between the name of the key and the name of the function.

StreamGroup

The StreamGroup class represents a group of device streams to allow more succinct declarations of device schemas when using multiple instances of the same device. The class also implements the __iter__ method, which allows us to remove the special case for classes which was present in the implementation of compositeStream.

We are adding a variable argument list in the constructor to make it easier to both declare nested classes as streams, or reuse other existing stream classes in the composition.

Usage Examples

To illustrate how to use the new API, we contrast it side-by-side with example definitions taken from the current architecture.

Simple Streams

These are single stream schemas where each chunk is named by a pattern composed by the device name, followed by a standard suffix and file extension.

old API

def position(pattern):
    """Position tracking data for the specified camera."""
    return {"Position": _reader.Position(f"{pattern}_200_*")}

new API

class Position(Stream):
    """Position tracking data for the specified camera."""
    def __init__(self, pattern):
        super().__init__(_reader.Position(f"{pattern}_200_*"))

Stream Groups

These are reusable logical groups of schemas applied to a single device. They are meant to be used as macros or templates for a composition of streams which is usually repeated across a large number of devices in a specific experiment.

old API

def environment(pattern):
    """Metadata for environment mode and subjects."""
    return _device.compositeStream(pattern, environment_state, subject_state)

new API

class Environment(StreamGroup):
    """Metadata for environment mode and subjects."""
    def __init__(self, pattern):
        super().__init__(pattern, EnvironmentState, SubjectState)

Stream Group Classes

The contents of reusable stream groups can also be specified by defining nested stream classes.

old API

class TaskLogic:
    @staticmethod
    def trial_initiation(pattern):
        return {"TrialInitiation": _reader.Harp(f"{pattern}_1_*", columns=["trial_type"])}

    @staticmethod
    def response(pattern):
        return {"Response": _reader.Harp(f"{pattern}_2_*", columns=["wall_id", "poke_id"])}

new API

class TaskLogic(StreamGroup):
    def __init__(self, path):
        super().__init__(path)

    class TrialInitiation(Stream):
        def __init__(self, pattern):
            super().__init__(_reader.Harp(f"{pattern}_1_*", columns=["trial_type"]))

    class Response(Stream):
        def __init__(self, pattern):
            super().__init__(_reader.Harp(f"{pattern}_2_*", columns=["wall_id", "poke_id"]))

Summary by CodeRabbit

Release Notes:

New Feature: Introduced Stream, StreamGroup, and Device classes for enhanced data stream management.
Refactor: Updated the Device class to handle multiple data streams and create streams based on provided arguments.
Deprecation: Certain usage patterns have been deprecated with warnings introduced for these cases. Users are advised to update their code accordingly.
Change: Altered signatures of exported functions and global data structures, which may affect external interface and behavior. Please review your integration if you're using these interfaces.

Please note that these changes involve significant modifications to the code structure and functionality.

SainsburyWellcomeCentre / aeon_mecha