Function would convert from avro/parquet/csv/json to avro/parquet/csv/json - Githubissues

luminousmen / data-toolset

Upgrade from avro-tools and parquet-tools jars to a more user-friendly Python package.

MIT License

2 stars 0 forks source link

Function would convert from avro/parquet/csv/json to avro/parquet/csv/json #6

Open luminousmen opened 1 year ago

luminousmen commented 1 year ago

As part of the tool, we need to develop a versatile data conversion functionality. As a result, the tool should be capable of converting data between various formats, including Avro, Parquet, CSV, and JSON. The goal is to provide a seamless and efficient way to transform data from one format to another to support our diverse data sources and consumption needs.

AC:

Create a data conversion library that supports the following conversions:

From Avro to Parquet
From Avro to CSV
From Avro to JSON
From Parquet to Avro
From Parquet to CSV
From Parquet to JSON
From CSV to Avro
From CSV to Parquet
From CSV to JSON
From JSON to Avro
From JSON to Parquet
From JSON to CSV

It should support existing data sample files from the tests/data directory
Write unit and integration tests to validate the correctness of the function.
Integrate the function into the interface
Ensure the function is easy to use with a clear and well-documented API.

ankan-mazumdar commented 12 months ago

Hello luminousmen, I would like to contribute, could you please schedule a meeting to explain the project.