paris-saclay-cds / ramp-workflow

Toolkit for building predictive workflows on top of pydata (pandas, scikit-learn, pytorch, keras, etc.).
https://paris-saclay-cds.github.io/ramp-docs/
BSD 3-Clause "New" or "Revised" License
68 stars 43 forks source link

caching #286

Open kegl opened 2 years ago

kegl commented 2 years ago

It happens sometimes that we call the same feature extraction on all submissions and that it takes a lot of time. We could use the idea here to cache the partial results. Doesn't seem straightforward since it's the user submission that should cache it's fit or transform, and I'm not sure how it will go through importing that file.

https://scikit-learn.org/stable/modules/compose.html#caching-transformers-avoid-repeated-computation