search

kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
84 stars 76 forks source link

Replace eager loading with lazy loading in `IncrementalDataSet` #250

Open jmholzer opened 1 year ago

jmholzer commented 1 year ago

Description

In tech design 21/07/23 we decided to implement lazy loading in IncrementalDataSet. See the comment under https://github.com/kedro-org/kedro/issues/2456. A prerequisite for this is that it is first migrated to kedro-datasets (https://github.com/kedro-org/kedro-plugins/issues/249).

PtrBld commented 1 year ago

I originally designed the pull request for the main repository, so I would start working on these two issues (#249 and #250) here now if that is ok for you?

jmholzer commented 1 year ago

I originally designed the pull request for the main repository, so I would start working on these two issues (#249 and #250) here now if that is ok for you?

That's great, thanks @PtrBld !

astrojuanlu commented 8 months ago

This is ready (finally!) to be done here on kedro-datasets, feel free to cherry-pick https://github.com/kedro-org/kedro/pull/2497