The whole tonic disk cache thing really didn't seem to work well for us so I've created an example of how you might cache tonic a dataset in memory (tonic datasets are stored about as efficiently as possible) and apply augmentations every epoch. @tnowotny if you want to tweak the augmentation I've implemented here to better match the best augmentations you've found, feel free! With models that train as quickly as this one does, augmentation does add quite a large overhead but I don't there's much way around this while everything's done in Python.
Also, I found a subtle bug that meant training epoch-by-epoch learnt very poorly - optimiser step was being reset to zero each epoch meaning that Adam made too large steps
The whole tonic disk cache thing really didn't seem to work well for us so I've created an example of how you might cache tonic a dataset in memory (tonic datasets are stored about as efficiently as possible) and apply augmentations every epoch. @tnowotny if you want to tweak the augmentation I've implemented here to better match the best augmentations you've found, feel free! With models that train as quickly as this one does, augmentation does add quite a large overhead but I don't there's much way around this while everything's done in Python.
Also, I found a subtle bug that meant training epoch-by-epoch learnt very poorly - optimiser step was being reset to zero each epoch meaning that Adam made too large steps