Prebatched dataset saving for particular tests

InfluenceFunctional / MXtalTools

BSD 3-Clause "New" or "Revised" License

7 stars 1 forks source link

Prebatched dataset saving for particular tests #125

Open InfluenceFunctional opened 5 months ago

InfluenceFunctional commented 5 months ago

uses more disk but speeds up training initialization

InfluenceFunctional commented 2 weeks ago

Currently, we load a generic dataset and filter & process it on the fly. We could speed up modelling for 'popular' tasks by pre-building and pre-saving task specific datasets.