search

idiap / importance-sampling

Code for experiments regarding importance sampling for training neural networks
Other
321 stars 60 forks source link

Remove unnecessary loading of validation data to memory #23

Open angeloskath opened 5 years ago

angeloskath commented 5 years ago

In ImportanceTraining classes we unnecessarily load all the data in memory.

https://github.com/idiap/importance-sampling/blob/991230a86d6d3d3a1d8b1a84716ae0db92248e26/importance_sampling/training.py#L288

MasterScrat commented 4 years ago

This is problematic - it makes validation completely impossible in some cases, eg when using generators that load heavy images! Is there any way to get around this?

angeloskath commented 4 years ago

I agree it is problematic and it needs to be changed. The simplest solution I can think of is to do an external for loop and evaluate the unwrapped model. Something like the following:

model = .....
wrapped = ImportanceTraining(model)
for i in range(epochs):
    wrapped.fit(...)
    model.evaluate(...)