search

libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.79k stars 180 forks source link

Varying output shape between batchs #313

Open AvivSham opened 1 year ago

AvivSham commented 1 year ago

Hi All, How are you? I wonder if FFCV supports flexible output shapes. For example, when dealing with audio files it's common that different batches will have different file lengths. e.g. batch 1: (N, 22000) | batch 2: (N, 16000) does FFCV support such case?

andrewilyas commented 1 year ago

Try using https://docs.ffcv.io/api/fields.html#ffcv.fields.BytesField !

AvivSham commented 1 year ago

hi @andrewilyas, thank you for your response do you have a tutorial/example of using this field? lets my batch is numpy array and its shape is (N, d) where d may change from batch to batch. I have a custom dataset that I would like to convert to .beton. Do I need to convert this numpy array to BytesArray when returning it in the __getitem__ func?