Open ElenaKhaustova opened 3 weeks ago
How does this relate to the existing transcoding functionality? (https://docs.kedro.org/en/stable/data/data_catalog_yaml_examples.html#read-the-same-file-using-different-datasets-with-transcoding) And for the Ibis
part, would that go beyond the IbisDataset that we've added recently?
Description
Users express the need for functionality to convert between different dataset formats at the catalog level. Additionally, integrating Kedro with existing standard dataset formats like
dlthub
andIbis
would provide users with a convenient way to work with diverse datasets and enable the seamless conversion between formats.We propose to:
CSV
,JSON
,Parquet
, and others, providing users with flexibility in working with diverse datasets.dlthub
andIbis
, allowing users to leverage these formats directly within the framework.Context
DataCatalog
is seen as a well-designed and battle-tested system that could greatly benefit from more integration with external ETL tools likedlthub
. This would allow users to leverage the strengths of these tools within the Kedro environment, enhancing data format transformations and interoperability without needing to develop extensive new dataset implementations.