search

duckdb / duckdb_iceberg

MIT License
107 stars 18 forks source link

Iceberg REST Catalog Support #16

Open randypitcherii opened 9 months ago

randypitcherii commented 9 months ago

Hey, team!

Very excited about the duckdb v0.9 support for iceberg!

I currently use a rest catalog for my iceberg tables and was hoping to be able to wire up duckdb to that rather than point it to the actual underlying data/metadata files.

If this is available, I'd love to use it -- otherwise, I'd be happy to jump in and start coding if this feature is new.

Thanks!

samansmink commented 9 months ago

Hi @randypitcherii

Thanks for your interest! The iceberg extension is currently in a quite early stage. The REST catalog is not yet supported, so we are definitely interested in your help there! Feel free to reach out to me through the DuckDB discord for a chat!

randypitcherii commented 9 months ago

Ok, no worries.

I'm thinking I'll chat with the rest catalog through python then get the details to my 🦆 db programatically.

I'll see you on the discord!!! Thanks!

thinkORo commented 3 months ago

Good morning @samansmink , is there any plan to support iceberg catalogs in general (not only REST) in the near future?

Thanks in advance.

samansmink commented 3 months ago

Hey @thinkORo! I would love to, but I'm a bit low on time currently. In general i would say we would like to support the most used catalogs at some point, but I can not give any timeline here at the moment. If you are interested in contributing, I'm happy to help out though

thinkORo commented 3 months ago

Hi @samansmink ,

Unfortunately, I'm only really good at Data Management and Data Analytics. And Python. Therefore, I am only a very limited support in contributing to DuckDB.

But: If I can do something to increase the prioritization or support you elsewhere to give you more time for such an (really important, at least for me) implementation, I am happy to do so.

rustyconover commented 2 months ago

I have a framework in place for this if #51 gets merged, see the notes about the REST/Nessie catalog.

It should just be a few more lines of work to perform the HTTP request.

astronautas commented 1 month ago

Up! Any updates on this?

rustyconover commented 1 month ago

Not yet -- been working on other things but will return to this soon.

arnabneogi86 commented 2 weeks ago

please update once it is implemented. Really excited to see duckdb support to REST catalog in iceberg