Video data preprocessing pipeline

[shikharbhardwaj]

Preprocessing incoming video data to produce input for the learning algorithm:

Requirements

1. Configurable Downsampling (resolution, FPS, color space)
2. Frame correlation (timestamps)

[shikharbhardwaj]

Data modelling

Downsample Config

@dataclass
class DownsampleConfig:
    height: int
    width: int
    col_space: ColSpace
    fps: float

@dataclass
class FrameMetadata:
    pts: PresentationTimestamp
    file_uid: str

@dataclass
class FileMetadata:
    uid: str
    filename: str
    metadata: Dict[str, str] # Arbitrary file-level metadata (eg: Map, Clip Type, etc)

Questions

1. How do we represent contextual clues?
2. Do we store these at the frame level or file level?

[shikharbhardwaj]

P0 implementation

• [x] DownsampleConfig
• [x] get_downsampled_frames

P1 implementation

• [ ] Add tensorflow/pytorch dataset adapters to load data for inference

Design overview