Connectors to the third-party services

[andrewyakonyuk]

There are a couple of ways Vault can obtain data from 3rd party API:

• [ ] Polling - repeatedly hit a REST endpoint looking for new data.
• [ ] Web hooks - "user-defined HTTP callbacks". They are usually triggered by some event. The same connector might have support of both of ways and these describes at connector level.

All these options are very similar in general, so it would be nice to provide the common mechanism to build a new connector or modify existing.

In general it can be looks like each connector provide a strategy to deal with this provider. That means, the strategy exposing a set of steps, which should be processed to obtain data. For instance: a. build request b. call 3rd party API c. parse the incoming data. This is absolutely valid for Polling strategy, but if API wants to notify the service itself, there are not need to build request and call him.

Connector life circle:

1. setup the connector for user. This might includes: a) schedule a next run b) register a webhook c) register a notification
2. prepare request (optional)
3. make a call (optional)
4. parse a response. (in this step the connector converts an incoming data into activity commands.)

Connectors API supports a different obtain data strategies (polling, webhook etc) which invokes any methods from life circle of the connector.

[andrewyakonyuk]

So, we have various types of 3rd party services, and each of them might provide his own specific logic of configuring itself and have an internal mechanism of obtain and deliver data to the sources.

That is why it is hard to extract the minimal core to achieve the goals:

1. provides a flexible API, so the developers will feel free with a new integration of connectors in future.

[andrewyakonyuk]

One of biggest the issue is creating a scheduler that will handle all time-related functionalities, inclusive of polling connectors.

Goal of such scheduler is in processing as much as possible of the polling requests per unit of time. In very optimistic scenario many users might be registered at the web service, and many of them have connected to the different social services. That is why the direct approach is not fit in the case. Under direct approach I am understand a scheduled task that will walk through all the users and their social connection and making requests to the appropriate API.

I propose to divide all API requests into queue according to the frequency of providing new data. Lets imagine, there are 4 queue:

0. near-real time requests processing. A requests for this group are makes every 5 minutes. The target audience is only users that is currently online on the website. Such users wants to see his own infomation as soon as possible.

The others 3 groups differ only in that how often they can provide some data.

1. At least once an hour
2. At least once a day
3. Not more often than once a day.

All these stuffs requires accounting and distribution requests system per queue. Lets say, the result of request execution will be evaluated and defined to the appropriate queue for next execution time.