search

rly / ndx-pose

NWB extension to store pose estimation data
BSD 3-Clause "New" or "Revised" License
13 stars 11 forks source link

Notes from discussion regarding training input #8

Open CBroz1 opened 2 years ago

CBroz1 commented 2 years ago

We plan to ...

  1. Extend ndx-pose - check backwards compatibility
  2. Assume external filepaths for images/videos
  3. Check in with Mathis team regarding this development
Pose Training input:
  Training parameters: json unspecified
  Skeletons:
    1+Skeleton:
      id: string
      1+Node Labels: string
      1+Edges: optional
  Training Frames:
    1+Training Frame:
      Annotator: optional string
      Source Video: optional
        Path: relative filepath
        Frame Index: unsigned int
      Source Frame: optional filepath
      Instances:
        1+Links to Skeleton: id
        1+Node Positions:
          x: unsigned int
          y: unsigned int
          z: optional unsigned int
CBroz1 commented 2 years ago

I posted to the DLC gitter to ask for input

roomrys commented 2 years ago

Proposed refactoring of pose training structure:

  1. Removed training parameters
  2. Add attribute for node visibility
  3. Add Identity group
  4. Add link to instance identity
Pose Training input:
  Identities: optional
       1+Identity: string
            name: string
            description: string
  Skeletons:
       1+Skeleton:
             id: string
             1+Node Labels: string
             1+Edges: optional
  Training Frames:
    1+Training Frame:
        Annotator: optional string
        Source Video: optional
             Path: relative filepath
             Frame Index: unsigned int
        Source Frame: optional filepath
        Instances:
             1+Links to identity: name
             1+Links to Skeleton: id
             1+Node Positions:
                  x: float
                  y: float
                  z: optional float
                  visible: bool
CodyCBakerPhD commented 2 years ago

@roomrys mostly LGTM; the only thing I'd suggest is for Source Video and Source Frame, these should be ImageSeries (which allows the user flexibility to choose between external vs. internal storage) and RGBImage data types, respectively.

With our default approach of writing the image frames directly to h5py.Dataset, both of those should work as-is for now but leaves it open for long-term flexibility for other user-chosen methods of data storage.

roomrys commented 2 years ago

Forked repo here with these specifications. Currently adding tests in SLEAP for writing/reading the proposed format. Subject to change and open to recommendations/concerns!

CodyCBakerPhD commented 2 years ago

https://github.com/talmolab/ndx-pose-training/blob/main/spec/ndx-pose.extensions.yaml is looking good - should we open a draft PR to ndx-pose to continue the discussion about any specifics there?