Bulk PATCH support

[Dragomir-Ivanov]

Currently rest-layer doesn't support bulk PATCH/UPDATE on list resources. This is not something standard in HTTP, nor REST world, and it seems that each user invents something on their own. My proposal is simple(for now) support for bulk PATCH(patch is what I need for now), that can be implemented in rest-layer:

PATCH on some resource list with following structure:

[
  {
   id: "YYY",
   etag:"XXX",
   patch: {<JSON-Patch here>}
  },
  ...
]

,with HTTP response with following structure

[
  {
    id: "YYY",
    etag: "ZZZ",
    status: <HTTP status code here>,
    response: {<HTTP response body here>},
  },
  ...
]

This will honor HTTP Prefer request header, so response might be missing patching is okay. If etag is supplied on the request it will act as If-Match HTTP request header, and will be checked. I guess we can add something similar for If-Unmodified-Since.

The reponse structure will include HTTP status code for each PATCH operation, and possible response field with either patched resource, or error payload.

Note that there is no atomicity of the whole operation, and it can't be guaranteed even if we had DB transactions, because hooks can execute arbitrary logic. I guess with the idea of HOOK middle-ware can improve things. Client is responsible to workout the situation if it needs atomic bulk operation.

This is just a proposal, and I will be glad to hear if there is any other good solution to bulk operations in REST world.

[smyrman]

Google Cloud have some generic support for batch requests for their APIs. A solution for you could be to have a separate end-point for batch requests where you decode them into sub-requests:

• https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch
• https://godoc.org/github.com/jfcote87/google-api-go-client/batch

This minimizes the number of HTTP requests, but not the load against the db.

Other good options for more specialized needs, involves writing RPC calls for your specific needs that operate against the same database as rest-layer. We have been doing this with success using the JSON RPC 2.0 spec and the zenrpc server-side library. JSON RPC also supports batch requests, and this can be enabled when relevant.

• https://github.com/semrush/zenrpc

[Dragomir-Ivanov]

Thanks @smyrman for the speedy reply! I hope you are well.

Just a quick note, there is a difference between bulk and batch operations. Bulk does repetitive action on a single resource, but batch can do all kinds of actions against different resources.

Now, I have looked at Google way, but I view it as overkill for my needs at the moment. Also I would like to keep things rest-layer way as possible, so no multi-part HTTP requests. Also I guess it would be good if rest-layer provides such(however limited) functionality out of the box. Regarding your way of external operationagainst the same database, I guess then you validate the request item against the schema, update Etag and updatedAt fields to keep rest-layer consistency. It seems a lot of work. Aren't you working against the same rest-layer resources?

[smyrman]

Indeed, there is a difference between bulk and batch.

The JSON bodies you describe looks more like what you might want to do for BATCH; especially with multiple return statuses embedded in JSON. From a design perspective, supporting BATCH is very simpe, and very powerful.

Bulk howver, is very tricky, because the design of it is so tricky. If I where to implement patch for the list view, I think I would have implemented it as a single patch to apply to multiple resources matching a filter. This is tricky in rest-layer because the patch cannot be validated independently of what it's actually going to patch. If it is to contain a completely different request and response structure to everything else, then it's not a good fit.

Back to the RPC bit, then yeah, we parse from regular structs into a map[string]interface{} and then use the resource layer directly, similar to how it's used by the rest-layer rest package to validate the final insert/update.One huge benifit of this, turned out to be that structs are very easy to work with (and copy), so it was relativly easy to implement API versioning, header-based version selection and backwards compatibility. This is not something we are able to do for our REST API requests!

RPC is not intended for pure CRUD operations though; we use it mostly for M2M applications that are sporadically updated, and where backwards compatibility is key. However, because it's JSON, we are able to use JSON RPC in the front-end as well for everything that is a bit more complex than plain CRUD requests.

[Dragomir-Ivanov]

Thinking from another perspective, with HTTP/2 multiplexed connection, I don't see a benefit from supporting BATCH exclusively. One can just fire multiple concurrent HTTP requests, and the browser will multiplex them into a single already prepared connection.

I guess for BULK patching, you are talking patching against an arbitrary filter, where I am referring to patching against specific items, with theirs ID specified. In any case, I would prefer some generic approach, rather RPC mechanism outside rest-layer. Versioning I guess can be supported with multiple rest-layer index-es, mounted on different routes, say /api/v1/,/api/v2/, etc.

I think for now I might just use multiple requests. Thanks @smyrman

[smyrman]

For inspiration, here is another design for BULK (from jsonapi.org):

• http://springbot.github.io/json-api/extensions/bulk/