API: Batch queries with transactions support and error handling.

[cebe]

follow up on #303

[samdark]

POST /batch could accept JSON of the following format:

[
    "type": "DELETE",
    "url": "/article",
    "data": {
        "ID": 123,
    }
],
[
    "type": "POST",
    "url": "/user",
    "contentType": "json",
    "tempID": "%userID%", 
    "data": {
        "name": "Alex",
    }
],
[
    "type": "POST",
    "url": "/article",
    "data": {
        "created_by": "%userID%",
    }
],

Here we're passing tempID "%userID%" to the batch. That means ID of the created record will be assigned to a placeholder. Later we're using it in another request.

Response should give back responses in the same order as requests were made:

[
    {"success": true, "id": "1234"},
    {"success": true, "id": "1235"},
]

Response content is to be defined (I'm not sure the info above is enough).

[dizews]

@samdark

Response should give back responses in the same order as requests were made:

More convenient way is to mark each request and response.

[samdark]

@dizews you mean like the following?

{
    'r01': [
        "type": "DELETE",
        "url": "/article",
        "data": {
            "ID": 123,
        }
    ],
    'r02': [
        "type": "POST",
        "url": "/user",
        "contentType": "json",
        "tempID": "%userID%", 
        "data": {
            "name": "Alex",
        }
    ],
    'r03': [
        "type": "POST",
        "url": "/article",
        "data": {
            "created_by": "%userID%",
        }
    ]
}

and

{
    'r01': {"success": true, "id": "1234"},
    'r02': {"success": true, "id": "1235"}
}

[dizews]

@samdark exactly.

[samdark]

Makes sense.

[bwoester]

I always though batching would only apply to requests of the same type. Like reading a batch of users. Inserting a batch of products. ...

Doing it this way looks way too much as tunneling to me. When do you start batching? Once you have 50 commands in your queue? 10, 5, 2? If you once implemented batching, why not using it all alone? It would still work with single messages, wouldn't it? An it wouldn't require a second implementation dealing with single messages. Only that we're basically doing something more SOAP-like now.

I might be wrong with this, but it just does not feel right to me. Is this a common approach for batching? Any other frameworks taking this route? Or any books/ articles proposing such a solution?

[samdark]

@bwoester batching is usually used when application is able to work offline so it accumulates commands till it's online when it sends all these to batch. I've used it in practice and, indeed, it's used for everything.

[dizews]

@bwoester Like Any POST request a batch query do not use http cache. It is one of reason to use single GET request.

[dizews]

@samdark I think batch response item should be a wrapper around a single response.

{
    "r01":{"status":201,"data":{}},
    "r02":{"status":200,"data":{}}
}

by this way we can have more flexible container. We can add a headers section or something else

[samdark]

@dizews makes sense.

[bwoester]

Seems like Facebook uses a similar approach.

However, Google on the other hand simply uses multipart/mixed messages for their batches. What I like about their approach is: you don't need to introduce new JSON structures (resembling http messages, containing headers, body, ...). Instead, you simply use an already existing mechanism to group your messages.

https://developers.google.com/storage/docs/json_api/v1/how-tos/batch

Also, we could batch transparently in the background: just intercept sent messages and depending on how many chatting is going on, group them and send as multipart/mixed (or custom batch format), or send the intercepted message as is.

[samdark]

Google's way seems to be a bit more mature. I like it.

[dizews]

+1

[Ragazzo]

what about parsing the response on the client ? If using google's approach then too much unneeded parsing need to be done by client (compare to simple json_decode and so on), its syntax also confusing and cant be used easily in debug . Also another question is how to get request body , it is unavailable from get / post, should we read php://input instead ? I like @dizews suggestion about request and json format

[mushahidh]

I have a scenario where 4 tables needs to be created and updated simultaneously. I am using transactions for creation where id of one table is foreign key of another table. But i found no solution for update . I wish this feature will be part of Yii ASAP.

[decimoseptimo]

+1 This seems being on hold for a long time, any plans for this?

[samdark]

Not for the next release unless someone from community will take care of it.

[decimoseptimo]

Damn, not skilled for the task for the time being.

[muet84]

hey, Sorry if posted on wrong place. I am newbie with yii2 api, I tried but not able to get how to call and make batch processing works.

If my url is abc.com/message-log POST and process single record is working fine, and can load records with GET on abc.com/message-log but as provided above abc.com/message-log/batch respond 404

Please let me know how to process batch records.

[samdark]

@muet84 because it's not implemented.

[papppeter]

Hi,

i created some kind of solution, it is based on google api batch solution. I would appreciate your comments.

still in first release. i need to do some testing, but it works for me.

https://github.com/p4it-kft/yii2-rest-batch