Usually it makes sense to implement loading routines very early in a data pipeline. But this can make it very dangerous to stack multiple pipeline elements on top of each other, as filtering out intermediate views will not results in less loading operations, but only hides them.
Enter the late loading capabilities of edflow: Instead of implementing a get_example method, which e.g. returns a dict of image, we can now implement a method, which returns a function used for loading the image of a certain example. Setting the flag dataset.expand = True will then evaluate all returned functions in the dict returned by the dataset. This flag is automatically set by edflow, right before passing the dataset to make_batches.
Usually it makes sense to implement loading routines very early in a data pipeline. But this can make it very dangerous to stack multiple pipeline elements on top of each other, as filtering out intermediate views will not results in less loading operations, but only hides them.
Enter the late loading capabilities of edflow: Instead of implementing a
get_examplemethod, which e.g. returns a dict of image, we can now implement a method, which returns a function used for loading the image of a certain example. Setting the flagdataset.expand = Truewill then evaluate all returned functions in the dict returned by the dataset. This flag is automatically set byedflow, right before passing the dataset tomake_batches.