WIP DNM datalake: something workable with Snowflake Open Catalog - Githubissues

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

https://redpanda.com

9.69k stars 591 forks source link

WIP DNM datalake: something workable with Snowflake Open Catalog #24176

Closed andrwng closed 2 days ago

andrwng commented 5 days ago

A few hacks to get something usable with Snowflake and Open Catalog.

uses TLS for REST client. TODO: very hacked together, and forces a futurized catalog factory
adds base (e.g. /api/catalog) and prefix components (my_catalog) to HTTP client. TODO: my_catalog is the "warehouse" component that should probably be configurable per topic?
TODO: create namespaces before creating table

Backports Required

[ ] none - not a bug fix
[ ] none - this is a backport
[ ] none - issue does not exist in previous branches
[ ] none - papercut/not impactful enough to backport
[x] v24.3.x
[ ] v24.2.x
[ ] v24.1.x

Release Notes

andrwng commented 2 days ago

Superseded by Michal's changes