search

datacontract / datacontract-cli

CLI to manage your datacontract.yaml files
https://cli.datacontract.com
Other
484 stars 94 forks source link

Add support for importing from iceberg table definition #452

Closed paulcichonski closed 1 month ago

paulcichonski commented 1 month ago

Import side of #449 .

I'm not going to have time for writing up the export side of this for a few days (at least) so just going to start with the import side and see what folks think. I think the export side is also a bit more involved since it is necessary to parse definitions into nested struct types of the iceberg schema.

I've mostly copied the model of the Spark import for this since the Iceberg table format is quite similar to the Spark table definitions. Eventually it might be nice to add support for ingesting the table definition directly from the catalog (via the pyiceberg sdk), but I did not attempt to tackle that in this first iteration. I'm guessing I missed things so please just let me know what needs to be changed.

This assumes the table definition is in JSON format as defined by https://iceberg.apache.org/spec/#appendix-c-json-serialization.

paulcichonski commented 1 month ago

re: tests, I get a lot of local test failures when running from the main branch so I haven't been able to confirm all tests are passing, but I don't think my changes should impact any tests other than the ones I added directly.

jochenchrist commented 1 month ago

Thank you for your contribution. Looks good to me.