takikadiri / kedro-boot

A kedro plugin that streamlines the integration between Kedro projects and third-party applications, making it easier for you to develop end-to-end production-ready data science applications.
Apache License 2.0
36 stars 6 forks source link

Support dataset factories #4

Closed takikadiri closed 9 months ago

takikadiri commented 11 months ago

kedro introduced dataset factories in 0.18.12 . Which uses pattern matching to reduce the number of catalog entries.

The datasets are matched/resolved lazily at load time, which don't make them natively compatible with kedro-boot that compile everything relative to the catalog before the first pipeline run, and threfore before the first datasets loading.

Currenly kedro boot support it through this contribution but we should go further by aligning with how kedro handle dataset factories and implementing it at catalog compiler level