Generator support?

[alex-jap]

Is there any possibility of using a generator in the data function for datasets that don't fit in memory? Keras' fit_generator method doesn't seem to be used anywhere. Any suggestions on how this could be achieved?

[Avsecz]

Hey, I started implementing it a while ago, but haven't finished due to time constraints. It's definitely something on my roadmap.

How this can be achieved?

The idea is to allow the data function to return a tuple of generators instead of loaded datasets:

import itertools 

def data_fn(a=1, b=10):
    def mygen(a):
        for i in range(a):
            yield i

    train = itertools.cycle(mygen(a))
    train.steps_per_epoch = a # pass the number of samples per epoch

    valid = itertools.cycle(mygen(b))
    valid.steps_per_epoch = b

    return train, valid

One would need to update the code at all the positions where model.predict, model.evaluate gets called with a conditional statement checking whether we are dealing with an iterator or a full dataset:

• https://github.com/Avsecz/keras-hyperopt/blob/master/kopt/hyopt.py#L355-L361
• https://github.com/Avsecz/keras-hyperopt/blob/master/kopt/hyopt.py#L391-L398
• https://github.com/Avsecz/keras-hyperopt/blob/master/kopt/hyopt.py#L46-L52

Feel free to send a PR in case you'd like to do it.