Add a config reload REST endpoint for a side car config reloader

[samgriek-arc]

Details

In order to update configuration of a delta share server it must be restarted. This works OK, but it would be very convenient to be able to implement a side car container to "watch" the config file and trigger a reload endpoint.

Architecture

How it works

Expose an endpoint on the delta sharing server which when called, reloads the config from disk. In Kubernetes this may be achieved by calling a SIGHUP command but the same could be achieved by calling a REST API so it would be available more universally.

For example, Prometheus provides several management API endpoints https://prometheus.io/docs/prometheus/latest/management_api/

1. A process or human modifies the config in place
2. The side car service or container sees the config file was updated and calls the 'reload config' REST endpoint
3. The server reads the config from disk

Request details

This request is just to add the management API endpoint /-/reload

[linzhou-db]

Hi @samgriek-arc , Thanks for the proposal, I'd like to understand more context to the request, are you starting your own delta sharing server? what server configs do you usually change?

[samgriek-arc]

@linzhou-db - We have a delta sharing server and it's currently running on AWS's ECS. We would like to manage the configuration more dynamically. We are currently evaluating Delta Sharing but it looks like the right choice so far.

One use case is around access control. For example we could generate a token for a user through a tool in our webapp. That token should then be used by our client to access the shares we've prepared for them. We would like for this to work automatically so we would write the new token to the config, trigger a hot-reload of the config, and then share the token with the client. We would deploy an array of delta sharing servers so each client has their dedicated instance. Another use case could be to create a share via a process triggered by a user which they would immediately want to explore. There are probably a lot of other use cases one could imagine.

[linzhou-db]

@samgriek-arc Thanks for the info! It makes sense. I've created a ticket for us to track this. Though we are on a tight schedule with tons of tasks, we'll discuss the priority and get back to you soon, stay tuned.

[nishith107]

@linzhou-db : Are there any efforts to move configuration such as share info and access control info to some external source such as mysql which is more dynamic as compared to file based approach ?

[linzhou-db]

@nishith107 Exactly, moving such config to database and build a server on top of it to allow configuration gives the user much better control on what's shared. Though, the oss server is more like a prototype server for developer's reference. Feel free to extend it as needed.

Or you are welcome to use databricks unity catalog for the exact functionality you are looking for.