[!NOTE] This ticket aims at suggesting a structure of the API in the long term, particularly when the Data Connector Server will be its own server, separated from the Private Data API (fetching any data, LLM, search, etc.).

In the short-term, we can take shortcuts and leverage the data API endpoints instead of having dedicated endpoints (for the logs for instance)

Suggested long-term API structure

Considering:

BASE_URL: "http://domain.com" API_URL: ${BASE_URL}/api/rest/v1

(Using /rest to be future-proof, in case a /api/graphql endpoint is necessary)

Data Connections API endpoints

All these endpoints are available under API_URL

Providers

GET /providers

Get all the provider definitions, include the handler definitions

Provider

GET /providers/:providerId

Get a given provider definition

Connections

GET /connections

To get all the connections. Could be a POST as the query (filter, sorting, pagination) is better in the body than in query/search params.

GET /connections/watch

Event Source endpoint. Get updates on all the connections, particularly theirs status. Could be a POST as the query (filter, sorting, pagination) is better in the body than in query/search params.

POST /connections/sync

Sync all the connections at once

GET /connections/logs

Get all the existing logs. Could be a POST to pass a query in the body.

GET /connections/logs/watch

Event Source endpoint. Listen to new logs for all connections. Could be a POST to pass a query in the body.

Connection

GET /connections/:connectionId

Get a connection object by its id

PUT /connections/:connectionId

To update the configuration of the connection. Not all properties of the connection object can be updated though, so the controller will have to carefully extract the allowed field from the body.

DELETE /connections/:connectionId

Disconnect a connection

POST /connections/:connectionId/watch

Event Source endpoint. To listen to updates on the connection

POST /connections/:connectionId/pause

To pause a connection

POST /connections/:connectionId/resume

To resume a paused connection

POST /connections/:connectionId/sync

To trigger the sync operation of the connection. Doesn't wait for the sync to finish to respond, doesn't return anything apart from whether triggering the sync was successful. The actual result of the sync can be checked via the status of the connection (also the /watch endpoint to listen to changes on the status)

GET /connections/:connectionId/logs

Get the logs of a given connection. Could be a POST to pass a query in the body.

GET /connections/:connectionId/logs/watch

Event Source endpoint. Listen to new logs for a given connection. Could be a POST to pass a query in the body.

Data Connections other routes

These routes are available under BASE_URL, not API_URL (/api/rest/v1) because they are not API endpoints, but are actual pages opened by the browser.

/providers/:providerId/connect

To connect to a provider using their authentication/authorisation mechanism (OAuth)

/providers/:providerId/callback

Passed to the provider OAuth mechanism to get the information back and creates the Connection object

Short-term needed updates

In the short-term, the /connections, the /connections/logs, the /connections/:connectionId, and the /connections/:connectionId/logs endpoints are not needed as the generic /db/query and /db/get endpoints can be used instead.

The /.../watch endpoints to listen to changes would be nice but not a priority as we can refetch the data occasionally.

The /providers/:providerId is also not needed as there is not a lot of providers, so /providers is enough for now.

The current GET /sync sure works, even though a POST /connections/:connectionId/sync would be more explicit and more RESTFUL.

Same for the GET /provider/disconnect/:provider, which works but would be more RESTFUL with as a DELETE /connections/:connectionId.

The only actual current need is the ability to update a connection as well as pause/resume it. When updating we have to be careful on what should be excluded from the update (providerId, handlerId, status, etc.) and what can be updated (basically just the config/options).

verida / data-connector-server

Restructure the API endpoints #112