search

apache / iceberg-go

Apache Iceberg - Go
https://iceberg.apache.org/
Apache License 2.0
143 stars 35 forks source link

feat: add support for catalogs with glue implementation to start #51

Closed wolfeidau closed 9 months ago

wolfeidau commented 10 months ago

This PR adds an implementation of the catalog which is heavily inspired by iceberg-python, I have tried to keep it as minimal as possible to start somewhere, and get feedback.

This includes coverage for all the glue related operations which can be easily mocked, with some discussion needed around testing of the s3 access.

I am doing my best to follow the conventions in the existing code, with a few things needing discussion:

  1. The table identifier is a bit confusing, I have done my best to follow the python structure in the catalog.
  2. Need to do a bit of work around testing combined glue and s3 access, maybe start by exposing a way to load the s3fs directly so we can pass in a mock to test LoadTable.

I can easily use this as a basis for the dynamodb catalog, which would enable running both the iofs and catalog locally in an integration test environment at some point.

wolfeidau commented 10 months ago

@nastra @zeroshade not sure if either of you saw this, would love some feedback.

wolfeidau commented 10 months ago

@Fokko hopefull that changes makes things a bit clearer, I have added more internal documentation as well on the interface to be more specific on behavior around these identifiers. 😅🤞

zeroshade commented 9 months ago

@wolfeidau Can you add check marks to the appropriate spots in the README for the functionality you're adding?

wolfeidau commented 9 months ago

@zeroshade done

nastra commented 9 months ago

thanks for the reviews everyone and thanks to @wolfeidau for getting this done!