Feature: Serverless compatibility => Attach to a database over httpfs or s3

ShravanSunder commented 3 weeks ago

Description

I think enhancing the HTTPFS functionality to allow attaching to remote databases, similar to DuckDB's implementation. This feature would enable direct, connections to databases hosted on platforms like S3 via HTTPFS. This would be great for serverless environments.

Currently, Kuzu’s HTTPFS support is limited to remote import/export functions. Mounting s3 is not feasible in serverless env.

Benefits

Generally it would be good to decouple storage from the database/compute.
Allows running in more diverse environments
Makes it easier to share populated databases

Links

DuckDB's guide on network and cloud storage: DuckDB over HTTPS or S3
Discord chats: https://discord.com/channels/1196510116388806837/1196510116871159909/1219866350722355220 https://discord.com/channels/1196510116388806837/1196510116871159909/1231212508572946483

giusedroid commented 3 weeks ago

yes please, I'd love this very much!

acquamarin commented 1 week ago

https://github.com/kuzudb/kuzu/issues/3465 I just wrote a design doc about how we are going to support this feature. Let me know if you have further questions.

acquamarin commented 2 days ago

Hi @giusedroid We just supported attaching a remote kuzu database(which can be stored in s3/https) in #3467. The feature is still evolving and we are working hard to improve the performance.

Looking forward to get your feedback!

kuzudb / kuzu