Support for multiple operations in a single request

tkellen commented 9 years ago

It has been widely requested that JSON-API support creating/editing/updating/deleting multiple records in a single request (#202, #205, #753, #536). We have attempted to paint this bikeshed repeatedly over the years. Providing an "official" solution, or solutions, to this problem is one of the primary goals for v1.1 of this specification.

There are several requirements for a solution:

It should allow mixed actions in a single request (e.g. creating + updating).
It should work for multiple types of resources (e.g. articles + tags) that would otherwise be accessed at different endpoints.
Every request should be transactional (i.e. all operations should succeed or fail together).
Additive to the base spec (probably as an extension).

Here are a couple use-cases that test these requirements:

Create two articles, add one existing tag to both, create a new tag and add to both.
Change the title of an article and update its tags: add one existing tag, create another, and remove another (without replacing the whole tag set).

We would like the community to propose ideas for how to solve these use cases (or chime in to support solutions provided in the responses to follow).

Assume the following data is present

GET /articles

{
  "data": [{
    "id": "article-1",
    "type": "articles",
    "attributes": {
      "title": "the first article",
      "content": "some content for an article",
    },
    "relationships": {
      "tags": {
        "data": [
          { "id": "tag-1", "type": "tags" }
        ]
      }
    }
  }]
}

GET /tags

{
  "data": [{
    "id": "tag-1",
    "type": "tags"
    "attributes": {
      "name": "one"
    }
  }, {
    "id": "tag-2",
    "type": "tags"
    "attributes": {
      "name": "two"
    }
  }]
}

dgeb commented 8 years ago

Potential Solution: Operations Extension

One solution to this problem is to represent each operation as an entity in a request document. Operations could be transmitted in an array, since ordering is often significant. Similarly, responses to each operation could be returned in an array that parallels the operations in a request.

This solution is very similar to the experimental JSON Patch extension with the exception that it is additive to the base spec.

Operations could include the following members:

path - the path to the applicable resource, if different from the request path
action - add / remove / update actions that map to HTTP methods
data - the data normally sent as top-level data

Operations would be sent in a top-level operations array.

Responses could also be returned in a top-level operations array, in which the status member could be added to each operation.

Requests would be sent with the PATCH method together with the TBD extension requirement.

It's important to mention that a server would have complete control over which operations could be performed together at any endpoint.

Solutions for use cases

Create two articles, add one existing tag to both, create a new tag and add to both

Request:

PATCH /articles

{
  "operations": [{
    "path": "/tags",
    "action": "add",
    "data": {
      "type": "tags",
      "attributes": {
        "name": "three"
      }
    }
  }, {
    "action": "add",
    "data": {
      "type": "articles",
      "attributes": {
        "title": "A second article",
        "content": "Lorem ipsum."
      },
      "relationships": {
        "tags": {
          "data": [
            { "id": "tag-1", "type": "tags" },
            { "pointer": "/operations/0/data" }
          ]
        }
      }
    }
  }, {
    "action": "add",
    "data": {
      "type": "articles",
      "attributes": {
        "title": "A third article",
        "content": "Lorem ipsum."
      },
      "relationships": {
        "tags": {
          "data": [
            { "id": "tag-1", "type": "tags" },
            { "pointer": "/operations/0/data" }
          ]
        }
      }
    }
  }]
}

Response:

{
  "operations": [{
    "status": "201",
    "data": {
      "id": "tag-3",
      "type": "tags",
      "attributes": {
        "name": "three"
      }
    }
  }, {
    "status": "201",
    "data": {
      "id": "articles-2",
      "type": "articles",
      "attributes": {
        "title": "A second article",
        "content": "Lorem ipsum."
      },
      "relationships": {
        "tags": {
          "data": [
            { "id": "tag-1", "type": "tags" },
            { "id": "tag-3", "type": "tags" }
          ]
        }
      }
    }
  }, {
    "status": "201",
    "data": {
      "id": "articles-3",
      "type": "articles",
      "attributes": {
        "title": "A third article",
        "content": "Lorem ipsum."
      },
      "relationships": {
        "tags": {
          "data": [
            { "id": "tag-1", "type": "tags" },
            { "id": "tag-3", "type": "tags" }
          ]
        }
      }
    }
  }]
}

Change the title of an article and update its tags: add one existing tag, create another, and remove another (without replacing the whole tag set).

Request:

PATCH /articles/1

{
  "operations": [{
    "path": "/tags",
    "action": "add",
    "data": {
      "type": "tags",
      "attributes": {
        "name": "three"
      }
    }
  }, {
    "action": "update",
    "data": {
      "type": "articles",
      "id": "1",
      "attributes": { "title": "Something more exciting!" }
    }
  }, {
    "path": "/articles/1/relationships/tags",
    "action": "add",
    "data": [
      { "pointer": "/operations/0/data" }
    ]
  }, {
    "path": "/articles/1/relationships/tags",
    "action": "remove",
    "data": [
      { "id": "tag-1", "type": "tags" }
    ]
  }, {
    "path": "/articles/1/relationships/tags",
    "action": "add",
    "data": [
      { "id": "tag-3", "type": "tags" }
    ]
  }]
}

Response:

{
  "operations": [{
    "status": "201",
    "data": {
      "id": "tag-3",
      "type": "tags",
      "attributes": {
        "name": "three"
      }
    }
  }, {
    "status": "204"
  }, {
    "status": "204"
  }, {
    "status": "204"
  }, {
    "status": "204"
  }]
}

tkellen commented 8 years ago

Leaving notes here for posterity, @dgeb and I discussed the possibility that we should consider supporting a response flag in operations with two allowed values, either full or status. This would control how detailed the response from the server will be for each operation. This would cover the case where the client does not care about the details of the intermediate representations created within a transaction.

We might also need to support a fetch action whose path can be a url or a pointer to one of the operations. This would allow a client to request a full representation of a json-api resource they care about (as it appears after all of the preceding operations were performed).

As an aside, this entire proposal feels like gross RPC, but there really doesn't seem to be a path forward to enable the complex use-cases we are trying to support without this type of extension.

jakerobers commented 8 years ago

This will probably get messy on the client side when you have relationships multiple layers deep. Doing multiple requests keeps data in a flatter structure, and will keep things simpler, in my opinion.

tkellen commented 8 years ago

This will probably get messy on the client side when you have relationships multiple layers deep. Doing multiple requests keeps data in a flatter structure, and will keep things simpler, in my opinion.

You can totally do that. We're discussing an optional/additive extension to the base specification that has been widely requested (see the issues referenced in the OP). If you don't want to use it, this issue doesn't apply to you.

mfpiccolo commented 8 years ago

I agree with @tkellen. Yes, using multiple requests keeps things simpler, however sometimes it doesn't make sense to do multiple requests especially if they are to be treated as a single transaction. It would be very difficult to manage a block of separate requests as a single transaction, rollback changes and return error states for the objects.

I am currently using included to handle these multi-operational actions which has its down sides and if I am rolling my own it means others are too. That is a good indication that we need a spec.

shicholas commented 8 years ago

I really like the proposed solution above. I am excited to see a standard arise out of this.

regarding this:

I discussed the possibility that we should consider supporting a response flag in operations with two allowed values, either full or status.

I like the idea of building an app that returns just the status if all my operations were update/delete, and full if any of the operations was an add. So I would like to see a response flag, perhaps as metadata?

and this:

We might also need to support a fetch action whose path can be a url or a pointer to one of the operations. This would allow a client to request a full representation of a json-api resource they care about (as it appears after all of the preceding operations were performed).

Does this mean that the server will return something other than the operations array? If so, I don't think it's a good idea b/c these resources can be included in that array already.

gr0uch commented 8 years ago

Not sure how I feel about this from an implementer's perspective. It adds a lot of complexity just to save on HTTP requests. That said, I'm glad it's being discussed as an optional extension.

dgeb commented 8 years ago

It adds a lot of complexity just to save on HTTP requests.

That is one primary goal. Another is to provide a means by which multiple operations can be performed in a single transaction.

And still another goal particular to the operations extension is to provide a format compatible with streams (e.g. web sockets).

tkellen commented 8 years ago

We might also need to support a fetch action whose path can be a url or a pointer to one of the operations. This would allow a client to request a full representation of a json-api resource they care about (as it appears after all of the preceding operations were performed). Does this mean that the server will return something other than the operations array? If so, I don't think it's a good idea b/c these resources can be included in that array already.

Does this mean that the server will return something other than the operations array? If so, I don't think it's a good idea b/c these resources can be included in that array already.

No, it means you might have a request like this:

{
  "operations": [{
    "path": "/articles/1/relationships/tags",
    "action": "remove",
    "data": [
      { "id": "tag-1", "type": "tags" }
    ]
  }, {
    "path": "/articles/1/relationships/tags",
    "action": "add",
    "data": [
      { "id": "tag-3", "type": "tags" }
    ]
  }, {
    "path": "/articles/1",
    "action": "fetch"
  }]
}

...with a response like:

{
  "operations": [{
    "status": "204"
  }, {
    "status": "204"
  }, {
    "status": "200",
    "data": {
      "id": "1",
      "type": "articles",
      "attributes": {
        "title": "the first article",
        "content": "some content for an article",
      },
      "relationships": {
        "tags": {
          "data": [
            { "id": "tag-3", "type": "tags" },
          ]
        }
      }
    }
  }]
}

shicholas commented 8 years ago

oh cool, I could see the utility in that. Thanks for the example.

ethanresnick commented 8 years ago

As an aside, this entire proposal feels like gross RPC, but there really doesn't seem to be a path forward to enable the complex use-cases we are trying to support without this type of extension.

That sounds like a fair diagnosis to me. That is, I agree that there are some complex cases that require transactional and imperative/RPC-ish semantics. Therefore, we'll need something reasonably like JSON PATCH, and @dgeb's proposal here seems cleaner than the current JSON PATCH extension.

However, I wonder if we could cover 80% of the embedding/sideposting use cases with a more declarative, higher-level format and, if so, whether that would make sense.

Sideposting Proposal

Overview

The higher-level other option I was thinking about is an extension, negotiated with the TBD extension mechanism, that would be something very similar to what I proposed with an "embedded" member in #536. The only differences are that it would:

be merged with the bulk extension
be extended to work with PATCH requests too
include more normative guidance on which types of relationship graphs must be supported, and a way to signal other graphs types that happen to be supported
offer a standard way to determine which embedded resource was assigned which server id (if server ids are in use).

Use Cases

Here's how it would work with the two use cases given above.

Creating two articles and adding existing and new tags to each:

Request:

POST /articles

{
  "data": [{
    "type": "articles",
    "attributes": {
      "title": "A second article",
      "content": "Lorem ipsum."
    },
    "relationships": {
      "tags": {
        "data": [
          { "id": "tag-1", "type": "tags" },
          { "pointer": "/embedded/0" }
        ]
      }
    }
  }, {
    "type": "articles",
    "attributes": {
      "title": "A third article",
      "content": "Lorem ipsum."
    },
    "relationships": {
      "tags": {
        "data": [
          { "id": "tag-1", "type": "tags" },
          { "pointer": "/embedded/0" }
        ]
      }
    }
  }],
  "embedded": [{
    "type": "tags",
    "attributes": {
      "name": "three"
    }
  }]
}

This is as I proposed in #536, except that I've taken advantage of the idea in the bulk extension to make "data" an array. The response would look exactly like the request, except that each resource object would now have a server-assigned id too.

Change the title of an article and update its tags: add one existing tag, create another, and remove another (without replacing the whole tag set).

This use case would need to be handled by operations, if it's to be done in one request. Otherwise, it would take 3 requests (which might be ok). One request would update the article's title; one would create the new tag and add it with the existing one; and one would remove the other tag. The interesting request is the one that creates and adds the new tag simultaneously, which would look (as you'd expect) like this:

POST /articles/1/relationships/tags

{
  "data": [
    { "pointer": "/embedded/0" },
    { "type": "tags", "id": "tag-2" }
  ],
  "embedded": [
    { "type": "tags", "attributes": { "name": "tag 3" } }
  ]
}

Specification Details

First, as I mentioned, this proposed extension would also include a way for the client to determine which embedded resources were assigned which ids (if server-side ids are in use). That would work like so:

In the request:

// …
"embedded": [
  { "type": "tags", "attributes": { … }, "temp-id": "1" }
]

Then, in the response:

// …
"temp-ids": {
  "1": "de305d54-75b4-431b-adb2-eb6b9e546014"
}

The "temp-id" key would be totally optional on the client's end, but the server would be required to send back the "temp-ids" mapping with any "temp-id"s it received. Alternatively, "temp-id" could be made mandatory and used in place of the JSON Pointers; I was in favor of that in #536, but am indifferent at this point. Also, if "temp-id"s are used, they might alternatively live/be returned in "meta", rather than at the top-level, depending on whether the extension system ultimately allows an extension to "claim" "meta" members.

About relationship graphs: the extension would be required to support graphs in which the only links between the resources in the request document are relationships from the primary data to embedded resources. (That is, the embedded resources don't link to one another, and the primary data's resource objects, if there's more than one, don't link to one another. But embedded resources can link to other, pre-existing resources.) This could be extended to say that any links are valid so long as the resulting graph is acyclic, to support the recursively embedded resources that @tkellen asked about on the original embedded resources proposal, in addition to interlinking between primary data resources, etc.

As far as supporting other graphs goes, the extension would have feature flags (again, mechanism tbd) indicating the types of graphs it supports. The structure of those flags would be defined over time as we collect more real-world use cases. A request might have a way to specify which type of graph it's sending, in order for the server to use more efficient processing.

Analysis

Pros, as I see them:

At least for some datastores, the higher-level syntax might be more performant and easier to implement transactionally than the raw "operations" format would be. In particular, I worry with the raw operations format that, unless the server analyzes the operations upfront to look for optimizations, it will simply apply them serially and in order, and that could be slow—especially, god forbid, if we allow some resource types to live across the network, as discussed here. By contrast, with the higher-level syntax, if we're just talking about must-be-supported graphs described above, a fast implementation is very simple: the server creates all the embedded resources at once/in any order and then creates/updates the primary data (at once/in any order) and sets their links to the newly-created embedded resources.

Or, if we expand the class of must-be-supported graphs to include all acyclic ones, the server treats the primary and embedded resources as one graph and creates them in the order returned by the well-known algorithms for topological sorting. (This also takes the burden of ordering the operations, which could be non-trivial, off the client, so long as it can guarantee that the graph is acyclic; that seems like a very good thing to me. For performance reasons, though, we could always add back the ability for the client to prespecify an order, if it knows a workable one in advance.)

In the sense that it has a clear processing model, this higher-level syntax is analogous to @dgeb's embedded resources proposal, though it supports a superset of that proposal's graphs and more closely mirrors the "included" feature by using an "embedded" array rather than nesting objects.
I think side-posting is easier to learn/use than operations, because the latter requires adopting a different mental model than the one used in the rest of the spec.
This syntax seems to cover almost all the cases we've been asked for directly: creating a post with its author, a credit card with its billing address, a conversation with its first message, an invoice with its line items, etc.
There's some chance that people would use the operations approach sub-optimally, e.g. by making all their operations to the root endpoint, which would give less-good HTTP caching than the side posting approach, in which the PATCH always goes through the appropriate resource/collection URI and therefore invalidates intermediate caches.

The biggest con I see to this higher-level syntax is that, because we'll probably still need an operations extension too, it's duplicative. However, if a good chunk of APIs could get away without implementing operations, and the high level syntax really has the advantages listed above (and doesn't have any unforeseen problems), then having it as an option might make sense anyway. After all, we shouldn't take the "it's duplicative" argument to its extreme, as that would suggest removing most of the base spec, since all one really needs is operations.

Proposal 2: Inline Operations

I think that the ideal request for the second use case would look something like the below:

PATCH /articles/1

{
  "data": {
    "attributes": { 
      "title": "New Title" 
    },
    "relationships": {
      "tags": {
        "operations": [
          { "type": "tags", "id": "tag-3", "op": "add" }, 
          { "type": "tags", "id": "tag-1", "op": "remove" },
          { "pointer": "/embedded/0", "op": "add" }
        ]
      }
    }
  }, 
  "embedded": [{
    "type": "tags",
    "attributes": {
      "name": "three"
    }
  }]
}

This only uses one request, but it feels less RPC-ish than the straight up operations approach, and it builds on the side posting syntax proposed above.

I think of this as the "inline operations" approach because it gets rid of "path" entirely in each operation and thereby tries to blend the operation ideas in with the existing JSON API spec. My hope is that an approach like this could make operations feel a bit more natural, but this is a very new idea, so I'm not sure if it'll actually work. I'm curious what y'all think!

shicholas commented 8 years ago

I like this suggestion a lot too. What happens if there are two levels of resources being created? e.g. "Creating two articles and adding new tag and tag category" That is a contrived example, and perhaps the spec should dissuade requests like that?

tkellen commented 8 years ago

The original draft of this post also included this use-case, but we didn't fill it out for time reasons:

Create this hierarchical data structure:

.
└── nodeA
    ├── aChild
    └── nodeB
        ├── bChild
        └── nodeC
            └── cChild

Transform the above structure to this:

├── nodeA
│   └── aChild
├── nodeB
│   └── bChild
├── nodeC
│   └── cChild
└── rootChild

GET /nodes

This actually starts empty, but the representation below matches this

.
└── nodeA
    ├── aChild
    └── nodeB
        ├── bChild
        └── nodeC
            └── cChild

{
  "data": [{
    "id": "nodeA",
    "type": "nodes",
    "relationships": {
      "parent": {
        "data":null
      },
      "children": {
        "data": [{
          "id": "nodeB",
          "type": "nodes"
        }, {
          "id": "nodeC",
          "type": "nodes"
        }]
      }
    }
  }, {
    "id": "nodeB",
    "type": "nodes",
    "relationships": {
      "parent": {
        "data": {
          "id": "nodeA",
          "type": "nodes"
        }
      },
      "children": {
        "data": []
      }
    }
  }, {
    "id": "nodeC",
    "type": "nodes",
    "relationships": {
      "parent": {
        "data": {
          "id": "nodeA",
          "type": "nodes"
        }
      },
      "children": {
        "data": [{
          "id": "nodeD",
          "type": "nodes"
        }, {
          "id": "nodeE",
          "type": "nodes"
        }]
      }
    }
  }, {
    "id": "nodeD",
    "type": "nodes",
    "relationships": {
      "parent": {
        "data": {
          "id": "nodeC",
          "type": "nodes"
        }
      },
      "children": {
        "data": []
      }
    }
  }, {
    "id": "nodeE",
    "type": "nodes",
    "relationships": {
      "parent": {
        "data": {
          "id": "nodeD",
          "type": "nodes"
        }
      },
      "children": {
        "data": [{
          "id": "nodeF",
          "type": "nodes"
        }]
      }
    }
  }, {
    "id": "nodeF",
    "type": "nodes",
    "relationships": {
      "parent": {
        "data": {
          "id": "nodeE",
          "type": "nodes"
        }
      },
      "children": {
        "data": []
      }
    }
  }]
}

If someone wants to take a stab at representing that in both proposed solutions, that would be great.

ethanresnick commented 8 years ago

@shicholas If I understand your question correctly, that would look like this:

POST /articles

{
  "data": [{ 
    "type": "articles", 
    "attributes": { "title": "New Article" },
    "relationships": {
      "tags": {
        "data": [ { "pointer": "/embedded/0" } ]
      }
    }
  }, {
    // second article would be just like the first one
  }],
  "embedded": [{
    "type": "tags", 
    "attributes": {
      "name": "new tag!"
    },
    "relationships": {
      "category": {
        "data": { "pointer": "/embedded/1" }
      }
    }
  }, {
    "type": "category",
    "attributes": { "name": "whatever" }
  }]
}

Per the earlier discussion, though, this example would only be supported if the extension signals it can create any acyclic graph or we make that a base requirement.

ethanresnick commented 8 years ago

@tkellen Re the other use cases:

Create this hierarchical data structure:

└── nodeA
    ├── aChild
    └── nodeB
        ├── bChild
        └── nodeC
            └── cChild

That would be done very similarly to my above comment, with a single POST to /nodes.

Transforming that structure, once created, though, would require two requests. The first would move nodeB and nodeC to the root level, pulling their descendants along with them:

PATCH /nodes

{
  "data": [{
    "type": "nodes", 
    "id": "nodeB", 
    "relationships": {
      "parent": { "data": null }
    }
  }, {
    "type": "nodes",
    "id": "nodeC",
    "relationships": {
       "parent": { "data": null }
    }
  }]
}

The second request would create the new rootChild node with a simple POST.

With inline operations, this might be able to be consolidated into one request, like so:

PATCH /nodes

{
  "data": [ // same as above ],
  "operations": [{ 
    "op": "create", 
    "data":  { 
      // new node resource object with parent null 
    }
  }]
}

However, the fact that the this complex a transformation takes only one or two requests feels like a lucky anomaly. It's enabled by /nodes happening to return every node (not just the root level ones). Moreover, we happen to be able to get by in this case without specifying the order in which the operations in data (or, in the consolidated version, data and operations) are executed.

For complex transformations like this to work in the general case, though, we'd probably have to specify that the updates in data are executed in order, and that the operations are run after the data changes are applied...which brings us back basically to the same processing model as operations. All of which is to say that there'd still be some cases in which just using operations makes sense, even if we have a higher-level alternative.

shicholas commented 8 years ago

Thank you @ethanresnick for answering my question.

FWIW, I like your proposal better than the operations approach because I feel it adequately addresses any POST use-case I intend on using with minimal changes to the base spec. I like how it keeps the data key, which makes what url I should send the requset to clear and makes what I feel the proper response should be (correct me if I'm wrong but I feel it would be the POST request described in the spec for the resource(s) described in the data key).

ethanresnick commented 8 years ago

I like how it keeps the data key, which makes what url I should send the requset to clear and makes what I feel the proper response should be (correct me if I'm wrong but I feel it would be the POST request described in the spec for the resource(s) described in the data key)

Thanks Nick. I was imagining the response would be the same as the current POST response, except that the newly-created resources from "embedded" would also be returned in the response. This allows the client to see the server-assigned id each was given, as discussed earlier, and any other server-set attributes.

ethanresnick commented 8 years ago

For reference, it looks like Facebook's approach to batch operations is to process all operations in parallel by default, but allow the user to specify dependencies between operations that should be executed in serial. See https://developers.facebook.com/docs/graph-api/making-multiple-requests

joananeves commented 8 years ago

Shouldn't this issue be marked with milestone ""JSON-API 1.1-beta"? "It's the primary feature on roadmap" as @tkellen said on Nested attributes aka. embeded records support that

ethanresnick commented 8 years ago

@joananeves yes! done. thank you

krainboltgreene commented 8 years ago

Okay, so I hope this doesn't come off bad (especially since I don't have a ready solution/alternative), but we're looking at this for my company and I came to the conclusion that this feels like SOAP over PATCH.

You can create, mutate, and delete resources under a single verb with this JSONPatch style body. That seems to be against the whole point of verbs. I hope I'm just reading this wrong.

dgeb commented 8 years ago

I came to the conclusion that this feels like SOAP over PATCH.

I have tried to ensure that the operations extension is a far cry from SOAP, which may use actions such as createArticle, addCommentToArticle, and approveArticle, and use completely custom payloads.

Instead, each operation has almost identical constraints to other JSON API compliant requests. Actions directly correspond to HTTP verbs. Essentially, the operations extension "steps back" one level to allow multiple JSON API requests to be performed together atomically.

You can create, mutate, and delete resources under a single verb with this JSONPatch style body. That seems to be against the whole point of verbs.

Actually, this concept of sending a set of instructions instead of a replacement resource is in keeping with the intent of PATCH:

With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version.

krainboltgreene commented 8 years ago

See that reads to me like the intent is for (as an example) sed like instructions. Two indications for me:

... a resource ...

... be modified ...

Neither of these phrases suggests the ability to create or destroy on a batch level.

I get the need to do a batch of requests. It's important for a large number of reasons that have been detailed here (thus my reluctance to say something), but it seems to be overreaching.

Granted I've never really been a fan of JSONPatch in the first place as it basically defines a special struct for something that could reasonably be represented as a transactions/ resource (where attributes are op, action, etc).

If we can create|update|destroy 4 articles with this extension, what's the point of BULK or collection based PUT?

dgeb commented 8 years ago

There's a distinction between HTTP resources and JSON API resources. One HTTP resource may encompass a large number of JSON API resources.

PATCH allows for a number of modifications to a single HTTP resource. From JSON API's perspective, these modifications may involve creating / updating / deleting any number of JSON API resources addressable as an HTTP resource.

If we can create|update|destroy 4 articles with this extension, what's the point of BULK or collection based PUT?

Are you referring to the bulk extension? It has a much narrower focus than the proposed operations extension. I'm not really sure if we should continue to propose it as an alternative to the operations extension because it is less expressive and less capable.

krainboltgreene commented 8 years ago

One HTTP resource may encompass a large number of JSON API resources.

Yeah, okay, I get that you can essentially do join tables in a server endpoint, but we're talking about mutation operations on multiple individual resources.

I just come to the feeling that if you can implement every possible interaction with an HTTP server over PATCH with JSONPatch-style operations, you've basically ignored the spirit of the protocol.

My finally suggestions for this thread are:

A transaction resource, in the same vein as error objects (that is a protocol that is significant enough to get it's own official definition)
A guide to streaming HTTP requests over a single socket.

ethanresnick commented 8 years ago

A transaction resource

I like this. To me, POST /transactions is more RESTful than PATCH /. The most reasonable interpretation of the / resource (in the HTTP sense) is, imo, "the API's entrypoint", not "the entirety of the API's data". To see this, consider that an operations request probably shouldn't invalidate any cached version of GET /, which I'd imagine would just hold JSON-HOME style entrypoint links. But, if the operations requests are made with PATCH /, rather than POST /transactions, they do invalidate GET / caches.

Of course, in the ideal case, you're making the operations request to something narrower, like PATCH /articles. But, for those cases where an appropriate, narrower URI doesn't exist, having some dedicated resource other than / does seem preferable.

Also, POST /transactions means that the server could presumably (if it wanted) return a handle to the transaction (with the id) to make it very easy for the client to retry or specify rolback.

A guide to streaming HTTP requests over a single socket

What do you mean here?

EDIT just to clarify my comment in light of @tkellen's below (and idk if this is what @krainboltgreene had in mind): all I'm saying is to send the request to a different URI; I'm not imaging any changes to the payload itself.

tkellen commented 8 years ago

@krainboltgreene Can you provide the sample payloads you'd propose to satisfy the use-cases outlined in this issue?

dgeb commented 8 years ago

I just come to the feeling that if you can implement every possible interaction with an HTTP server over PATCH with JSONPatch-style operations, you've basically ignored the spirit of the protocol.

Are you saying that JSONPatch itself ignores the spirit of the HTTP protocol? If so, then I disagree. However, I'm sure we'd all agree that JSONPatch is not traditionally "RESTful".

I think that "transactions" is a bit of a misnomer, especially if more than one can be POSTed in a request. The spec already requires that every request must be completed as a single transaction (i.e. succeed or fail completely), so I prefer the name "operations" for multiple individual actions taken within a request.

Naming aside, I've already tried my hand at modeling the concept of operations within the constraints of the current JSON API spec - see https://gist.github.com/dgeb/3dc80490208ef8e50586 (note: not proposing this alternative, just throwing it out there as something similar to @krainboltgreene's suggestion).

Ultimately I decided to propose the above operations extension instead, because it allows for isolation of the results of each operation but still allows clean cross-referencing of one result from another. Definitely worth exploring alternatives here though - just giving my reasoning :)

To me, POST /transactions is more RESTful than PATCH /

If we're allowing for the creation of resources at both /articles and /tags in the same request, whether it is directed to /transactions or /, then we are conceptually addressing multiple "JSON API resources" at a single endpoint. No matter where the relative path for operations is introduced, it seems to be required somewhere.

ethanresnick commented 8 years ago

If we're allowing for the creation of resources at both /articles and /tags in the same request, whether it is directed to /transactions or /, then we are conceptually addressing multiple "JSON API resources" at a single endpoint.

@dgeb Right. I'm not opposed to creating/updating multiple (JSON API) resources with a single request; if that's not allowed, then no proposal is going to work! :)

All I'm saying is that we get a tiny bit more RESTfulness by making a separate URI to direct these requests to than we do by overloading /, which I think of as meaning "the API's entrypoint". By making a separate URI, we don't unnecessarily invalidate the cache for /, which is important for true hypermedia clients (of the future) that start every request by requesting it.

The request to this separate URI would still use the same operations payload you've proposed and get operations results back. I haven't thought about how this would effect the path (I think my preference would be to always make it absolute, so that this would have no effect), but that's not hard to solve.

Also, we wouldn't have to dictate what this separate URI is—the extension could just provide the client with a link to it—though we'd probably make a recommendation. If we did make a recommendation, I agree that /transactions wouldn't be the right name, since all requests are transactional. What we're really talking about here is a "multi-resource PATCH", so maybe the URI could be /multi-resource-patches?

dgeb commented 8 years ago

I think we're getting into territory where we really miss discovery services (tentatively targeted for 1.2).

I say this because we've successfully avoided getting into the URI game thus far, and discovery services done right could allow us to continue to do so.

I imagine that some APIs might want to support the operations extension (or equivalent) only at certain endpoints, and might really restrict which operations can be performed at each endpoint. For instance, articles and tags could be created at /articles, but not other completely unrelated resources could not.

Alternatively, I could also see another API providing a single endpoint for all operations (whether it be / or /operations).

I think that providing for discovery of capabilities at endpoints is the right answer to make both approaches viable.

tkellen commented 8 years ago

:+1: to that. I'm really :-1: on the notion of us recommending a URI, especially one that might be in use already like /transaction.

krainboltgreene commented 8 years ago

For what it's worth I don't think the proposed solution by @dgeb is going to like, destroy the world or anything. Ultimately as an optional extension it's up to the server to conform. That server could have it's own extension for these things (bulkgraph maybe?).

My opinion is that in the end jsonapi.org is going to have to make a third class of specification: The Best Practice Specification, essentially articles detailing how to do something successfully, but don't need "special case structure" (like errors, this proposal).

On Thu, Sep 17, 2015 at 1:11 PM, Tyler Kellen notifications@github.com wrote:

[image: :+1:] to that. I'm really [image: :-1:] on the notion of us recommending a URI, especially one that might be in use already like /transaction.

— Reply to this email directly or view it on GitHub https://github.com/json-api/json-api/issues/795#issuecomment-141212955.

Kurtis Rainbolt-Greene, Hacker Software Developer 1631 8th St. New Orleans, LA 70115

ethanresnick commented 8 years ago

@dgeb: I think we're getting into territory where we really miss discovery services (tentatively targeted for 1.2).

@tkellen: :+1: to that. I'm really :-1: on the notion of us recommending a URI

I think we're all in agreement here. That is, I'd rather not recommend a particular URI either, actually. (As I said in my prior comment "we wouldn't have to dictate what this separate URI is—the extension could just provide the client with a link to it".) But, I think we should recommend that servers use some separate URI of their choosing instead of /.

shicholas commented 8 years ago

This has been a very interesting thread to follow, I've definitely learned a lot.

I don't want to sound brash, but I am a little worried given how disparate @dgeb and @tkellen's solution is to @ethanresnick's and the impending 1.1 deadline the collaborators seem to have at the end of the month, a spec may not surface in a couple weeks? Please tell me I'm silly for thinking so.

And out of curiosity, how will this be decided upon?

ethanresnick commented 8 years ago

Please tell me I'm silly for thinking so.

At our last meeting, we decided to go with @dgeb's basic solution, just making sure that it was specced to give the server freedom to apply the operations in parallel by default. I think he was going to put together an updated PR for that for everyone to look at. (Right?)

I still think my proposed syntax has value, but it's basically "sugar" over @dgeb's sensible primitives, so it can always be added later, and it doesn't preclude the need for @dgeb's solution, since it doesn't cover every case.

If the 1.1 deadline slips, it'll probably be my fault because I haven't had time to thoroughly go back to #614 and consider all the extension negotiation stuff. But there's no real disagreement on this issue :)

shicholas commented 8 years ago

cool, thanks for the reassurance. :+1: And fwiw if the deadline slips, oh well; I'm very grateful you're spending the time to make this spec awesome!

dgeb commented 8 years ago

I think he was going to put together an updated PR for that for everyone to look at. (Right?)

Correct! I will try to get this worked up in the next week.

As a general FYI: nothing in the 1.1 beta will be set in stone until 1.1 final is published (target date: Dec. 31). During the beta period, we'll be encouraging implementers to try out the beta features and raise any issues. If some of the new features turn out to be impractical or problematic to implement, we'll consider going back to the drawing board. In extreme cases, we even have the option to drop a feature from beta and push it out until the next release cycle.

lolmaus commented 8 years ago

Ran into a necessity to implement transactional saving. When saving several records (different model types), all save operations should be canceled if one of the requests fails.

What is the status of this RFC? Please give an ETA, even the vaguest one.

Please have your 3000th star! :smile: :tada: :beers:

Imgur Imgur

dgeb commented 8 years ago

@lolmaus thanks for the :star:!

As for the status of this RFC: my apologies for being distracted. I will attempt to write up a PR for the operations extension within the next few weeks. This is important to many people - myself included.

Of course, we still need to sort out the extension mechanism as well for the operations extension to be usable.

I would hope that we can sort all of this out in the next month or two.

RuslanZavacky commented 8 years ago

We've following this one too, as its pretty important to have ability of sending relationships to server in one call. Meanwhile, is there at least close to the good solution, right now? How to overcome this problem? How you guys solve it, until we have correct way to do it? :)

andruby commented 8 years ago

Can I add a proposal when creating a resource together with its relationships? We already have the perfect distinction between resource identifiers and resource objects. Why not simply allow resource objects inside the relationship data field?

{
  "data":{
    "type":"orders",
    "attributes":{
      "email": "jeff@example.com"
    },
    "relationships":{
      "line-items":{
        "data": [
          {
            "type":"line-items",
            "attributes":{
              "title":"The Ember Book"
            }
          },
          {
            "type":"line-items",
            "attributes":{
              "title":"The Ruby Book"
            }
          }
        ]
      }
    }
  }
}

Actually, it felt so natural that I already built this into our backend and was surprised that Ember-Data, nor the JSONAPI spec support this.

lolmaus commented 8 years ago

@andruby That's already implemented in Ember with the EmbeddedRecordsMixin (though it might have issues with JSONAPI).

On the JSONAPI side, embedded records are suggested as a custom extension on the Extensions page.

andruby commented 8 years ago

@lolmaus Thanks for the tip on Ember's EmbeddedRecordsMixin.

The fact that this is already supported in Ember, should give it added points for inclusion in JSONAPI 1.1, right? :smile:

KronicDeth commented 8 years ago

@andruby, @lolmaus I also read the spec as allowing resources (instead of resource identifiers only) in relationships on created and update.

frank06 commented 8 years ago

Any progress on JSON API 1.1 and support for multiple operations?

Petah commented 8 years ago

Any updates on this?

robconrad commented 8 years ago

+1 any progress here?

krainboltgreene commented 8 years ago

Honestly, those looking for a solution: Just use new resources. There's no need for a special type of specification. Resources can be whatever you want them to be, including a list of operations.

meshuga commented 8 years ago

@krainboltgreene :+1: Please, just make a FAQ/The Best Practice Specification or whatever that will explain how it can be achieved using existing specification and close this issue.

masterspambot commented 8 years ago

I have to agree with @meshuga. And it's becoming a bit troublesome for JSON API - we are trying to make almost every pattern standardized whereas we can simply just consume standard and apply patterns on base of that instead of trying to push everything into "standard way".

json-api / json-api