search

delta-incubator / delta-sharing-rs

A Minimalistic Rust Implementation of Delta Sharing Server.
MIT License
77 stars 8 forks source link

Introduce extension points for share storage #31

Open tdikland opened 5 months ago

tdikland commented 5 months ago

Currently the Delta Sharing server depends on a postgres database to store information about the defined shares, schemas, and tables. For a variety of reasons users may want to use a different store. My proposed solution if to introduce a ShareStore trait (open to naming suggestions) and let the app state depend on a concrete implementation defined in a library or user supplied.

tdikland commented 5 months ago

I'd be interested to work on this.

roeap commented 5 months ago

This makes a lot of sense!

In fact I am hoping, that we can introduce extension points for various components of the sharing server - including the share store.

Regarding the naming - while I don't really have a better name, conceptually this should mirror catalogues, such that we can for instance populate the shares and their children from unity catalogue ...

I don't really remember the UC APIs, but we may want to have the option to add some more descriptive metadata on top of this. Main application would be to be able to populate UIs o.a. with some information consumers may be interested in around the shared datasets.

tdikland commented 5 months ago

take

tdikland commented 5 months ago

Thanks for the feedback! Let me put up a draft and discuss further from there.

roeap commented 5 months ago

Not sure if this makes any sense at all, but I just had a thought 😆.

What if we more or less mirror the CatalogList, SchemaProvider, and TableProvider from datafusion - of course adjusting some types to out specific needs...

and then add some sort of metadata trait on top of this. This would allow us to re-use the StorageCatalog from delta-rs directly and potentially extend this to other catalogues that are implemented for datafusion.

Then again, just a quick thought ...

tdikland commented 5 months ago

I've put up a draft for now. Let me have a think about your suggestions and see how they would fit in here:)