Added support for HDF5 dataset and an HDF5 creation tool

fizyr / keras-retinanet

Keras implementation of RetinaNet object detection.

Apache License 2.0

4.38k stars 1.96k forks source link

Added support for HDF5 dataset and an HDF5 creation tool #1468

Closed madisi98 closed 1 week ago

madisi98 commented 4 years ago

Added retinanet-build-hdf5 entry point which allows the creation of datasets in the hdf5 format and a new option 'hdf5' to retinanet-train. This allows the dataset to be loaded in main memory the whole time and drastically reduces training times.

madisi98 commented 4 years ago

I haven’t made a lot of testing yet, since I haven’t had the time for it, but I was getting roughly 6x faster epochs. I have to say that my dataset is composed of large images (around 3000x2000) and with smaller images the speedup will be less significant.

hgaiser commented 4 years ago

I would be interested to see the differences on a more "normal" dataset like COCO. I expect the difference will be much smaller there because AFAIK the most time spent there is from anchor target generation, not data loading.