Conduct market research on versioning

astrojuanlu commented 3 weeks ago

In https://github.com/kedro-org/kedro/milestone/63 there are several linked issues related to Kedro's Dataset Versioning.

Before we start working on it, we'd want to do a bit of market research on other tools and formats that support versioning. At a minimum, it should include

MLflow, which can version inputs, models, and artifacts for experiment tracking (already well integrated with Kedro)
DVC, written primarily to version data and models using git, later expanded to include experiment tracking (there have been several attempts to integrate it with Kedro, see #2691)
Delta Lake and Apache Iceberg, which support time travel (Delta, Iceberg) for tabular data through SQL queries (Delta is already supported in Kedro, although with some caveats https://github.com/kedro-org/kedro-plugins/issues/542)

The objectives are

Assess the different ways these tools fulfill the goal of versioning,
Whether they treat different artifacts (data, models, generic outputs like images) differently (similar to how Kubeflow separates artifact types),
Clarify overlaps between the tools.

The end goal is to inform decision making around Kedro Dataset Versioning.

noklam commented 2 weeks ago

We should also review https://github.com/kedro-org/kedro/pull/1871

astrojuanlu commented 2 weeks ago

Notice that the goal of this is not to assess current Kedro versioning capabilities, but rather to provide an outward looking perspective at what other systems are doing. That ideally should inform next steps in https://github.com/kedro-org/kedro/milestone/63

kedro-org / kedro

Conduct market research on versioning #3933