Migrate dataloader to tensorflow dataset using .h5

renezurbruegg commented 3 years ago

I am really unhappy about the dataloader. I think the code is messy and will be hard to maintain :D.

I think it would be best to rewrite it and split it into two parts: 1) Tf dataset as done for the nyu labels, that uses a .h5 file and contains all information. 2) Short dataloader class that applies cropping, resizing, skipping of timestamps and data augmentation on top of the TF dataset.

This way it can also easily be transferred to different devices.

What do you think @hermannsblum? Related #18.

Related comment: I think it would be better to actually call this subsampling and specify how many frames to skip (makes it transferrable to different datasources independent of how long the rosbag is)

_Originally posted by @hermannsblum in https://github.com/ethz-asl/background_foreground_segmentation/pull/16#r556454983_

hermannsblum commented 3 years ago

I think it would make sense to basically have 2 things:

dataloader from a rosbag. This is required to generate data in the first place.
As soon as we require speed in the loading process, dump that data into a h5 and use the tfds-based loader since it has all the speed-related functionality.

hermannsblum commented 2 years ago

Should be fine now that we have all data loaders in either directory or gdrive format

ethz-asl / background_foreground_segmentation

Migrate dataloader to tensorflow dataset using .h5 #19