Design `DataCatalog2.0`

[merelcht]

Description

The current DataCatalog in Kedro has served its purpose well but has limitations and areas for improvement identified through user research: https://github.com/kedro-org/kedro/issues/3934

As a result of DataCatalog user research interview we have created the list of tickets and split them into 3 categories:

1. Tickets that require low or medium effort and can be implemented for the current DataCatalog version without introducing any breaking changes. (partially addressed)
2. Tickets with all effort/impact levels that should be implemented for the new catalog version - DataCatalog2.0. These tickets include significant catalog redesign steps before the implementation.
3. Tickets mostly with a high effort level which didn't fall into the previous item because they are touching some significant conceptual changes or new features that require a decision on whether we want to implement them.

Addressing issues from 2. and 3. requires significant changes and the introduction of new features and concepts that go beyond the scope of incremental updates.

The objective is to design a new, robust, and modular DataCatalog2.0 (a better name is welcomed) that incorporates feedback from the community, follows best practices, and integrates new features seamlessly.

While redesigning we plan for a smooth migration from the current DataCatalog to DataCatalog2.0, minimizing disruption for existing user.

Context

Suggested prioritisation and tickets opened: https://github.com/kedro-org/kedro/issues/3934#issuecomment-2153342972

Related topics

https://github.com/kedro-org/kedro-starters/tree/main/standalone-datacatalog https://github.com/kedro-org/kedro/issues/2901 https://github.com/kedro-org/kedro/issues/2741

Next steps

• [x] Diagram Current Design:

1. Create a detailed diagram of the current DataCatalog architecture.
2. Identify components, their interactions, and existing data flows.
3. Highlight pain points and limitations identified during user research.

• [ ] Analyze Similar Projects:

1. Research and document how similar projects (such as Unity Catalog, Polaris, dlthub, other ?) address similar tasks and challenges.
2. Identify features, implementations, and best practices from these projects and map them with features requested/insights obtained during user research.
3. Analyze their architectural approaches and note down pros and cons.

• [ ] Propose Initial Design:
  1. Based on the research and analysis, propose an initial modular design for DataCatalog2.0.
  2. Ensure the design addresses the tickets in group 2 and conceptual changes/new features from group 3.
  3. Use the initial design as a starting point for the discussion of the redesign process with tsc.

[datajoely]

One push here - and it's already addressed some what in your excellent write up @merelcht, is the fact that we should look to delegate / integrate as much as we can.

The data catalog was the first part of Kedro ever built, it was leap forward for us in 2017 but the industry has matured so much in that time. We should always provide an accessible, novice user UX... but I think now is the time for interoperability.

• Slightly controversial - I think we should make the first party endorsed patern Ibis-centric. It's portable to any back-end reducing our own maintenance burden, lazily evaluated, MICE and enables us to expose things like Column Level Lineage to our users.
• We look to delegate versioning/time-travel to modern table formats like Iceberg / Delta etc.
• We look to provide easy integration to modern catalog architectures e.g. Unity / Polaris. I would highly recommend reading this series by Chris Riccomini (1, 2, 3)

[ElenaKhaustova]

DataCatalog2.0 design board: https://miro.com/app/board/uXjVKy77OMo=/?share_link_id=278933784087

[datajoely]

Love the board @ElenaKhaustova there is an argument we should have all of those in PUML on the docs too 🤔 bit like #4013

[ElenaKhaustova]

Value Proposition and Goals for DataCatalog Redesign

The current DataCatalog implementation effectively addresses basic tasks such as preventing users from hardcoding data sources and standardising I/O operations. However, our goal is to enhance and extend its capabilities to reduce maintenance burdens, integrate seamlessly with enterprise solutions, and leverage the strengths of existing tools like DeltaLake and UnityCatalog. By keeping the datasets-based solution while integrating with UnityCatalog, we aim to cover a broader range of use cases and improve flexibility and functionality. With this redesign work, we aim to:

• Simplify the codebase to make it easier to maintain and address pain points identified at the user research interview for the current implementation.
• Ensure backward compatibility to avoid breaking existing workflows.
• Offer a mechanism for seamless integration with enterprise data catalogs like UnityCatalog (we chose it to implement POC as it serves ~50% of client use cases, though we would like to focus on the integration API so we can switch between different catalogs in the future).
• Start with ETL processes and extend integration to modelling workflows.
• Utilize top-n features from UnityCatalog to address common pain points such as versioning format conversion.
• Use integrated UnityCatalog features to run Kedro pipelines on various platforms.
• Maintain the ability to start smoothly without significant installation overhead.
• Retain the current datasets-based solution for user familiarity.

Strategy and Implementation Plan

Develop a flexible API and specific data catalogs, allowing switching between different catalog implementations and utilising their feature while keeping the current dataset-based solution with adjustments.

1. Design AbstractDataCatalog Implement an AbstractDataCatalog with common methods - exposed to framework: load(), save(), confirm(), release(), exists(), shallow_copy(), and __contains__() (subject to change?).
2. Create specific implementations for KedroDataCatalog and UnityDataCatalog. Common methods will be exposed to the framework. Methods not supported by a specific catalog will raise a "Not supported for this type of catalog" exception. KedroDataCatalog will keep the current dataset-based solution with the updates needed to address the pain points identified at the user research interview.
3. Define key features for Integration Learn the UnityCatalog API and how it aligns with its open-sourced part. Identify the top UnityCatalog features that will provide the most value for integration for POC.