Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
Several teams are already utilizing the catalog as a standalone component, demonstrating a demand for this functionality. Potential use cases may include collaboration between teams, sharing catalogs without the framework, and integration with other frameworks. Recognizing the existing adoption of the catalog as a standalone component highlights its potential value outside the framework's context.
We propose to explore the possibility of making DataCatalog a standalone component (moving it outside of the framework).
Data analysts already use datacatalog as a standalone component, they don’t even know about kedro pipelines : "Actually the the data analyst population I described at the beginning use 100% as a standalone component. We just have the Omega config loader at the beginning just to help creating the catalog. But then the users use it as a standard component and we do not introduce any other component. We have more users using the catalog as a standalone component than users using it with Kedro."
CSTs use DataCatalog as standalone component with Metaflow pipelines: "We were using Metaflow instead of the other software which is similar to Kedro from Netflix. They don't have catalog kind of API. So we used Kedro catalog with Metaflow back then with Hydra config loader."
User Demand and Manageability: There is sufficient user interest to justify making DataCatalog standalone. Managing it in a monorepo is feasible given current CI/CD setups.
Marketing and Adoption: Positioning DataCatalog as a separate component could serve as an entry point for users to discover and adopt the broader Kedro framework, enhancing user engagement.
Comparison with Similar Tools: Comparing it to other tools like Anaconda’s intake catalog, which hasn’t seen widespread adoption, suggests that while it’s not a top priority, there is a niche to fill.
Collaboration and Standardization: The standalone DataCatalog could improve team collaboration by standardizing how data resources are accessed and discussed, moving away from inefficient practices like sharing file paths directly.
Feature Expansion Potential: Adding functionality to read from and write to YAML could make DataCatalog a comprehensive, independent tool, increasing its utility and appeal.
Description
Several teams are already utilizing the catalog as a standalone component, demonstrating a demand for this functionality. Potential use cases may include collaboration between teams, sharing catalogs without the framework, and integration with other frameworks. Recognizing the existing adoption of the catalog as a standalone component highlights its potential value outside the framework's context.
We propose to explore the possibility of making
DataCataloga standalone component (moving it outside of the framework).Relates to https://github.com/kedro-org/kedro/issues/3659, https://github.com/kedro-org/kedro/issues/3932
Context
DataCatalogas standalone component with Metaflow pipelines: "We were using Metaflow instead of the other software which is similar to Kedro from Netflix. They don't have catalog kind of API. So we used Kedro catalog with Metaflow back then with Hydra config loader."DataCatalogstandalone. Managing it in a monorepo is feasible given current CI/CD setups.DataCatalogas a separate component could serve as an entry point for users to discover and adopt the broader Kedro framework, enhancing user engagement.DataCatalogcould improve team collaboration by standardizing how data resources are accessed and discussed, moving away from inefficient practices like sharing file paths directly.YAMLcould makeDataCataloga comprehensive, independent tool, increasing its utility and appeal.