Open alex-jap opened 6 years ago
Hey, I started implementing it a while ago, but haven't finished due to time constraints. It's definitely something on my roadmap.
How this can be achieved?
The idea is to allow the data
function to return a tuple of generators instead of loaded datasets:
import itertools
def data_fn(a=1, b=10):
def mygen(a):
for i in range(a):
yield i
train = itertools.cycle(mygen(a))
train.steps_per_epoch = a # pass the number of samples per epoch
valid = itertools.cycle(mygen(b))
valid.steps_per_epoch = b
return train, valid
One would need to update the code at all the positions where model.predict
, model.evaluate
gets called with a conditional statement checking whether we are dealing with an iterator or a full dataset:
Feel free to send a PR in case you'd like to do it.
Is there any possibility of using a generator in the data function for datasets that don't fit in memory? Keras' fit_generator method doesn't seem to be used anywhere. Any suggestions on how this could be achieved?