search

duckdb / duckdb_iceberg

MIT License
160 stars 23 forks source link

Iceberg REST Catalog Support #16

Open randypitcherii opened 1 year ago

randypitcherii commented 1 year ago

Hey, team!

Very excited about the duckdb v0.9 support for iceberg!

I currently use a rest catalog for my iceberg tables and was hoping to be able to wire up duckdb to that rather than point it to the actual underlying data/metadata files.

If this is available, I'd love to use it -- otherwise, I'd be happy to jump in and start coding if this feature is new.

Thanks!

samansmink commented 1 year ago

Hi @randypitcherii

Thanks for your interest! The iceberg extension is currently in a quite early stage. The REST catalog is not yet supported, so we are definitely interested in your help there! Feel free to reach out to me through the DuckDB discord for a chat!

randypitcherii commented 1 year ago

Ok, no worries.

I'm thinking I'll chat with the rest catalog through python then get the details to my 🦆 db programatically.

I'll see you on the discord!!! Thanks!

thinkORo commented 7 months ago

Good morning @samansmink , is there any plan to support iceberg catalogs in general (not only REST) in the near future?

Thanks in advance.

samansmink commented 7 months ago

Hey @thinkORo! I would love to, but I'm a bit low on time currently. In general i would say we would like to support the most used catalogs at some point, but I can not give any timeline here at the moment. If you are interested in contributing, I'm happy to help out though

thinkORo commented 7 months ago

Hi @samansmink ,

Unfortunately, I'm only really good at Data Management and Data Analytics. And Python. Therefore, I am only a very limited support in contributing to DuckDB.

But: If I can do something to increase the prioritization or support you elsewhere to give you more time for such an (really important, at least for me) implementation, I am happy to do so.

rustyconover commented 7 months ago

I have a framework in place for this if #51 gets merged, see the notes about the REST/Nessie catalog.

It should just be a few more lines of work to perform the HTTP request.

astronautas commented 5 months ago

Up! Any updates on this?

rustyconover commented 5 months ago

Not yet -- been working on other things but will return to this soon.

arnabneogi86 commented 5 months ago

please update once it is implemented. Really excited to see duckdb support to REST catalog in iceberg

buremba commented 3 months ago

While the combination of DuckDB <> PyArrow <> PyIceberg support covers this use-case, the extension is much more efficient than loading the data into PyTable. I would love to see the support for Iceberg catalogs.

prasanthkn83 commented 2 weeks ago

This integration is super exciting. Any updates on when we might expect it to be available? Looking forward to trying it out.