[DataCatalog]: Interface to request specific properties from catalog and datasets

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

Apache License 2.0

9.47k stars 875 forks source link

Description

Users seek increased efficiency in accessing data through the public API. They suggest implementing a kind of GraphQL interface that allows them to request specific properties from catalog and datasets, reducing unnecessary data transmission.

We propose to explore the feasibility of implementation and integration of such an interface into the public API.

Partially related to https://github.com/kedro-org/kedro/issues/143

Context

"If we want some information from catalog we would like to have kind of GraphQL interface where you ask for a few information, like let's say, passing as an argument and then it checks if that key is available and catalog itself resolves that and returns the details. So that it might be like a single interface or single API for interacting with catalog instead of customizing on top of it."
"If there is some interface where we ask for certain properties which we need from the catalog and catalog provides us those details instead of creating a session context and then getting a catalog."

kedro-org / kedro

[DataCatalog]: Interface to request specific properties from catalog and datasets #3939

Description

Context