libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.79k stars 180 forks source link

Problem while unpacking encoded JSONField #273

Open anasserhussien opened 1 year ago

anasserhussien commented 1 year ago

Hello,

I am using FFCV for an object detection task. When writing the .beton file, I am using RGBImageField for the image and JSONField for writing the dict of bounding boxes.

writer = DatasetWriter(path, {
    'image': RGBImageField(),
    'label': JSONField()
    }
)
writer.from_indexed_dataset(ds)

Writing the .beton file works properly with no problems. However, when I am looping over the batches after loading the .beton I am getting an error while unpacking the label field to get the json back.

for i, batch in enumerate(train_data_loader):
    x, y = batch

    #this line sometimes through an error
    y = JSONField.unpack(y)

The error says the following:

sep_location = np.where(b == ord(SEPARATOR))[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

Which basically means that the SEPARATOR does not exist in the encoded array. This happens only to few number of data points, any thoughts?