Expose collections via an HTTP endpoint

[stuartc]

Re: https://github.com/OpenFn/lightning/issues/2190

Provide an HTTP endpoint that exposes methods that calls functions inside the Collection context.

Streaming is a requirement of this feature, we should be using features like send_chunked and chunk (from Plug).

Auth

Access to this API must be authenticated using either a Users access token, or a RunToken, in the Authentication Header

API spec

GET /collections

Fetches/searches for values within a collection

GET /collections/{collection_name}/{key}

• Key is optional. A get on /collections/<collection_name> returns everything in the collection
• Key can include multiple wildcards * for pattern matching
• Query params:
  • updated_after / created_after only returns records GTE this the start of (last second? of) this date
  • updated_before /created_before only returns records LTE the end of (last second? of) this date
  • limit only return the first N records
  • offset search from this offset in the db
  • meta flag to include metadata (timestamps) in as a _meta key on each value
• Timestamps in queries are expected to be parsable as ISO 8601 strings.

Returns:

{
  results: [{ key, value, created, updated }],
  end_offset: 123
}

Each item is a { key , value, created, updated } object (I'd like a name for this wrapper - Record?)

Other metadata, like end_offset for paged results, is included at the top of the object as siblings of results

POST /collections

Upsert values into a collection

POST /collections/{collection_name}

• Values are uploaded as a JSON object in the body as an array of { key, value, ...meta } objects (where meta is not used now but would be a collection of metadata properties)
• Returns 404 if the collection does not exist
• Returns 201 if any new values were created
• Returns 200 otherwise
• New values will be assigned a created_at and updated_at timestamp. Existing values will have their updated_at changed.

Returns

For any 20x payload, return a JSON summary in the form:

{ upserted: 101, errors: [{}] }

DELETE /collections

Remove values from a collection

DELETE /collections/{collection_name}/{key}

• Key can include multiple wildcards * for pattern matching
• Query params:
  • updated_after / created_after only returns records GTE this the start of (last second? of) this date
  • updated_before /created_before only returns records LTE the end of (last second? of) this date

Returns

If possible, return a JSON summary in the form:

{ deleted: 101, keys: [...]}

[josephjclark]

@stuartc

I guess we'll use use Authorization header like BEARER <token> for all requests? Probably base64 encoded. Run tokens aren't sensitive, but when we add client API keys they will be.

[EDIT: pending our conversation in slack maybe it's /collections/<collection_name>?key=xxx I think GET needs to be like this:

GET /collections/<collection_name>/<key-pattern> 

Where key-pattern is optional

With query parameters (now or in future): offset, limit, since/after?, until/before?

I think the POST is always an upsert (if if a record exists, it'll be removed)

I have an idea that the post body is an array of the form { key: string, record: any, meta: any }. Meta would allow us to do things like override timestamps (maybe), or set expiration (perhaps?) per record. I don't know if we need it but we could use that as a bucket of system stuff.

What, if anything, should post return? An array with a status for each item? Maybe something more like { upserted: 101, errors: [] }

We also need a DELETE /collections/<collection_name>/<key>. Should we support a key pattern here or is that too exciting?

[josephjclark]

Updated the main issue body for GET/POST/DELETE requests, as per the Canvas conversation