Postprocessing is slow in general

thesofakillers / nowcastlib

🧙‍♂️🔧 Utils that can be reused and shared across and beyond the ESO Nowcast project

https://giuliostarace.com/nowcastlib

GNU General Public License v3.0

0 stars 0 forks source link

Postprocessing is slow in general #5

Closed thesofakillers closed 3 years ago

thesofakillers commented 3 years ago

Splitting data into test/train/val before the vast majority of our postprocessing seems unnecessary, and we actually end up making redundant computations this way. For example when generating new fields. Splitting is basically only necessary for standardization.

Should be in this order

preprocess
sync
postprocess
generate new fields
split
standardize

thesofakillers commented 3 years ago

Release 3.1.0 addresses this