Support URI templates

[dominicbarnes]

I've seen this exact issue mentioned several places and proposed as a solution. I'd like to formally express interest in adding support for this feature in Siren.

I've never personally used URI templates, at least not yet. Upon a quick search, I came across RFC 6570, but I'm sure there are alternatives.

[dominicbarnes]

I think the place this makes the most sense is in actions, where adding a URI template implies that variables will be replaced by the data being submitted with the action:

{
  "name": "create-resource",
  "method": "put",
  "href": "http://example.com/resource/{id}",
  "fields": [
    { "name": "id" }
  ]
}

Thus, submitting the action with { id: 'foo' } would result in:

PUT http://example.com/resource/foo

id=foo

I'm not sure if replacing a value in the URL should imply that it is excluded from the request body, but I'm inclined to leave it in both places.

[kevinswiber]

To be honest, the main reason this isn't in the spec is because it goes against my own personal API design guidelines in many cases. I treat parent resources like Aggregate Roots. Aggregate Roots own their child Entities for the specific context in which they're being accessed. When the need comes up for URI templates, I often take a look at how I'm modeling the parent Entity. Does this action actually belong on the top-level Entity instead of the child Entity? This usually buys me a few things. A big one is in analytics. Looking at the example action you presented, how will I know at the API server if a User Agent created a resource willy-nilly or if they actually executed the create-resource action? I really can't. Sometimes this is important to know. I suppose sometimes it's not.

In short, I'd like to see some real world usage before dropping it into the spec. I'm not opposed to adding it, even though it goes against my personal guidelines, as long as it's perceived as beneficial to the folks using Siren (other than me, of course).

[apsoto]

yea real world examples first please. I've personally tried, but it's one of those 'nice to have' that I never seem to find time for. And if there's no time for it, then obviously no one's asking for it.

[dominicbarnes]

Here's a real-world example from where I work:

When creating accounts, we use a slug that can be encoded into the URL instead of a UUID. (like you see on github, where I have my username that gets used in most URLs)

During signup, when people enter their slug, we check that against the existing accounts to make sure it is available. (we do it inline, rather than requiring a form submission) Prior to siren, we accomplished this by executing a HEAD /api/accounts/{slug} request. I was thinking about doing a similar thing with siren, rather than implementing a separate existence check endpoint.

[ChrisMarinos]

Here's a couple examples from an application our team is working on:

We're consuming our Siren API from a Single Page App in the browser. The client SPA has it's own URL scheme that may or may not correspond to URLs on the server. For example, navigating to /users/4 in the browser will cause the client to: 1) request the API's root URL 2) find the "user-api" sub-entity 3) find the "find-by-id" action on that sub-entity 4) execute the action by substituting "4" for the id field 5) display the user edit screen using the resulting user or display 404 if the user wasn't found.

In the above example, we use a GET action with a query parameter to generate a URL like /users/search?id=4. Currently, we have to build "search" URLs for every entity that we build in addition to URLs like /users/4 that we use for self links. It'd be nice if we could just tell our client how to construct the self link directly.

More problems arise when we introduce SingalR into the mix. Our client often wants to "observe" entities in the system, so we offer the ability for a client to subscribe to a SignalR topic to be notified when an entity changes. Just like in the basic HTTP example, we need to be able to generate these topic strings based on id. However, without the ability to use a template, we have no easy way to tell the client how to build a topic such as /users/{id}/changed. We could potentially solve this by introducing more requests - the server could have an action that the client could use to ask the server to build it a topic. However, this just seems unnecessary when we could just tell the client how to build it on it's own using a template.

Maybe there are other, better ways to solve both problems. If y'all have any ideas, I'm all ears. That being said, I think my team would definitely welcome templates into the Siren spec.

[apsoto]

Take with a grain of salt, as I have not implemented this. However, I envisioned the root url of my api having all the direct access urls accessible via links. I would provide all the resources with URI templates as shortcut links for easy deep dive into a resource when you have only an ID. The user and search rel below has some uri template params.

Example:

{
 links:   [
      {  rel:   [  "self" ],  href: "https://api.wurl.com/api"  },
      {  rel:   [  "me" ],  href: "https://api.wurl.com/api/users/me"     },
      {  rel:   [   "user"  ], href: "https://api.wurl.com/api/users/{id}"   },
      {  rel:   [   "search"  ], href: "https://api.wurl.com/api/search{?q}"  },

[ChrisMarinos]

That looks very similar to how our "ideal" approach would look. Our root api looks a little different since we have an embedded sub entity for each domain, but the net result is the same.

One of my team members reminded me of another complication to our use case that's worth mentioning for this discussion:

We have an in-memory cache of entities on the client. Our preference is to use the self link of an entity as it's key in this cache. Unfortunately, for the find-by-id case mentioned above, the client sometimes doesn't know the self-link-url of the entity before it requests it, and if the id 404s, it never knows the "self link". That means we have to use two different types of keys for each entity (the id and the self link). It sounds like a minor thing, but since we have to do it for every entity, it adds significant complexity to our mental model.

Giving the client the ability to build the full self-link-url would be a big win for us in terms of simplifying this caching story.

[kevinswiber]

@apsoto @dominicbarnes @ChrisMarinos

I hear ya. State requests (GETs) are really where this approach shines, I think. I think it might be easier to discuss in the context of a Pull Request.

@ChrisMarinos It sounds like you've got some skin in this game. Do you have time to take a first stab at a PR?

Thanks for the discussion, everyone. It's this kind of back-and-forth that really pushes the spec forward.

[apsoto]

@ChrisMarinos WRT the client side cache, it's not the server/spec's concern how a client manages its cache. That said, what's there to cache if the url 404s? I don't see the problem

[ChrisMarinos]

@kevinswiber Sure, I'll take a stab at a PR with the team here and try to come up with something in the next week or so.

@apsoto Definitely agree that the server/spec shouldn't care what the client does with it's cache. I was just saying that in our case, we have to associate types of URLs, so having an extra affordance from the server would help. Here's a workflow to help illustrate:

1. User points browser to http://mycoolsite.com/users/4
2. Client requests http://mycoolsite.com/api
3. Client finds sub-entity users and the find-by-id action on that sub-entity.
4. Client uses that action to make a request to http://mycoolsite.com/users/search?id=4
5. Server returns 404 (or 200 with an empty result set)
6. Client remembers that http://mycoolsite.com/users/search?id=4 does not exist
7. Someone else creates user 4
8. Client is notified of the change (through SignalR or some other mechanism), and issues a request to load the entity.
9. Client has to figure out somehow that the entity at http://mycoolsite.com/users/4 changes the state of http://mycoolsite.com/users/search?id=4 from "does not exist" to "exists".

Right now we handle the problem in part 9 by having the server send down id as a property for every element and tracking entity state using a combination of id and self link.

[apsoto]

IMO, 404, means no caching, you always request again. If you mean how does the server push that info to the client, then I think you are out of scope WRT siren.

[ChrisMarinos]

Calling it a "cache" might be the wrong way to phrase it as it implies HTTP/browser caching. I'm just referring to our JS app's in-memory state.

Either way, we could follow "404 means no caching", and we do for some things (e.g. search). I was just trying to illustrate some of our client-side workflows that we're finding cumbersome with Siren. We've already worked around most of our issues, but URL templates would help.

As you said above, they're a "nice to have" feature, but I like having nice things. :-)

[apsoto]

I'd love to keep discussing, take it to the google group

[M1strall]

Any updates about uri-templates? As a team we think it would be quite useful to have specification support as we already have them for actions implicitly

[xogeny]

I see not much has happened here since October. I wanted to jump in to add a few comments on this topic.

In my opinion, the "fields" for Siren are not that coherent. I think URI templates could actually solve a problem that Siren has but that most people perhaps haven't thought about or recognized. Let me explain.

I suspect almost every application of fields for actions use the url encoded approach. In this case, the fields are added to the URL to determine the URL that the request should be made from. This works well if you are dealing with HTML forms, etc. You could also specify something like application/json as the type for the payload in the case of a POST. However, in my experience this is of limited use because most of the time when I'm submitting a payload it is pretty complex and not easily described by a flat list of fields, but requires a schema of some sort. I think that is a topic worth discussing (and I've brought it up in the Google Group before), but it isn't what I want to focus on here.

The issue I see with fields (and how this related to URI templates) is that I think the current approach in Siren tries to abstract together two distinctly different things. Consider that you could and URI templates in this way with Siren today:

"actions": [
    {
      "name": "getItem",
      "title": "Find an instance in this collection",
      "method": "GET",
      "href": "http://api.x.io/orders/{id}",
      "type": "rfc6570", /* Not a real content type, I know...but see below */
      "fields": [
        { "name": "id", "type": "number" }
      ]
    }
  ]

With respect to the specification, the only issue I see here is that the spec says that href has to be a URI, which it isn't. But you could expand that to say that href needs to be a URI template and I think you'd be fine.

But there is a bigger problem here than just trying to shoehorn the URI templates in here and that is the issue with fields I mentioned.

A concrete example of this would be what if I wanted to simultaneously encode some fields in the URI and some in the payload. This is not an unreasonable thing to want to do. What if I wanted to POST some application/json to /api.x.io/orders/5 using the above example?

Some of the data is going into the request body and some is going into the URI. The reason I referred to these as distinctly different above is that the URI is meant to identify the resource whereas the request body is really the "data" for the request. As such, confusing these two things has implications for things like caching.

I actually think this is an example of how URI templates could really help clean up this confusion because a URI template is strictly a scheme for identifying the specific resource we are trying to address. If you allowed the href to refer to a URI template, then it would be clear which fields are being used to identify the resource (i.e., the variables named in the URI template) and which were part of the payload.

That would address the ambiguity, but my guess is that this doesn't address the whole issue. I say this because I would bet that in most cases, people using Siren actions almost always use the URL encoding option for type. In other words, they are almost always using fields to specify the identity of the resource. So if you used URI templates, you wouldn't really need type anymore, at least not for what it is currently intended to be.

In other words, I think actions really need a way of specify both what resource the request should be made to and what the form of the request body (if any) would be. As I've tried to argue, I see these as two distinctly different things that are currently muddled together. As such, I would propose to use URI templates strictly for resource identification and use fields to further document and constraint the variables in the URI templates and then, in addition, introduce some orthogonal way of describing the request body. My proposal, in the Google Group, was to include some kind of object that uses keys to represent accepted content types and values as a content type specific specification to the values...see below).

Under such a proposal, almost nothing would change for existing Siren actions that use URL encoding. In the case where type was URL encoding (the default case), all fields not explicitly mentioned in the URI template of the href could just be appended to the end (as they are now). But this would now not only eliminate the ambiguity about whether fields described the resource or the request body, but when combined with a separate scheme for specifying the request body, you could have Siren actions that were, in my opinion, much clearer. For example, consider these three very different requests that could all be expressed quite nicely (IMHO) under such a scheme:

"actions": [
    {
      "name": "getItem",
      "title": "Get an instance in this collection (i.e., self links for items in collection)",
      "method": "GET",
      "href": "http://api.x.io/orders/{orderNumber}",
      "fields": [
        { "name": "orderNumber", "type": "hidden", "value": "42" },
      ]
    },     {
      "name": "addItem",
      "title": "Add item to order (POST via query string...typical current Siren use I suspect)",
      "method": "POST",
      "href": "http://api.x.io/orders/{orderNumber}?{productCode}&{quantity}",
      "fields": [
        { "name": "orderNumber", "type": "hidden", "value": "42" },
        { "name": "productCode", "type": "text" },
        { "name": "quantity", "type": "number" }
      ]
    }, {
      "name": "addNewProduct",
      "title": "Add a new product (lots of structured data required)",
      "method": "POST",
      "href": "http://api.x.io/products",
      "fields": [], /* None because the resource we want is full specified */
      "type": {
        "application/json": { "$ref": "http://api.x.io/schemas/productSchema.json" },
        "application/xml": "<xs:include schemaLocation=\"http://api.x.io/schemas/productSchema.xsd\"/>"
      }
    }
  ],

The point here is that href and fields work together to specify the resource and type is used to specify a) which Content-Type values the server will accept for the request body and b) specify a schema for the request in a format that makes sense for the content type. For most schema types, it is possible to either inline the schema or, to save space, reference it from some external location in completely standard compliant ways.

Note that using URI templates for href would be 100% backward compatible because an ordinary URI is a completely valid URI templates (since a URI template is not required to contain any expressions). Also, URI templates are quite rich in what they can describe (e.g., use of the "explode" modifier makes it clear if a field can appear more than once). Also, there are standard libraries for handling URI templates, so the implementation burden wouldn't be that great.

Comments?

[apsoto]

Thanks for the detailed post.

My two cents. Early on in my design of an api, I thought URI templates would be great. I even stuck URI templates in some Link#href fields at the root level of the api. However, in my limited experience (one api implementation) with Siren I have not encountered the need for them with our clients.

[jopnick]

We've produce several API's using the siren spec, and I found we've been going out of our way to hide ID's wherever possible. A URI is an id, and an id with context present. Templating pushes the requirement of id spaces to the client, and forces 'pick-and-place' logic.

eg. IF (property.id) && url.hasTemplate -> than replace tokenWithId.

this compared to "if(hasLink) use it

I'd also wonder about what happens when you have multiple Id's, objectId with a fileId, userId etc. does the client map the property by name to templated marker? What happens if you really like to use property.id as your id

This problem is negated if the the id's are FQDN URI's; they are explicit as to their type/location and require no client knowledge

The above mentioned example I would think is a good one for a task-based workflow. Where you provide search to locate the entity in the collection and just store the URI.

In the case where the client needs to specify the ID, say, a guid, I think this would work the same way: you post the id field to the server, and it returns you the location of the newly created entity.

I guess the key question I have is: Where did you get the ID's from that you are wanting to use in the template?

My first assumption was that the entity.properties.id would be stored once the entity was created, and you are going to be inserting it to get the entity back again. Entity creation would normally 201 you to the location of the entity. Why not just store the self URI ? That is, instead of storing property.id to get it back again, why not just store the full href?

This is also true for sets, if you are getting a set of objects, instead of passing an array of ID's, why not just fetch them? Batching and bulk calls are normally the retort, and are perhaps a legit case, but when I can, I'd prefer to just batch via http/2 or pass the enum of URI's as a worst case.

They biggest issue we've seen is going back and forth between JSON api's and Hypermedia api's. JSON api's sure <3 id's and templating, and you get into a rather tight spot sometimes hiding the Id's. The transition back and forth between them has caused us to 'leak' id's as properties on occasion. We've still been able to work around templating as a first class thing, and my fear is making it part of the specification makes it to easy to do what traditional JSON RPC api's do, and perhaps perpetuates contextless id's (id's which aren't URI's)

[xogeny]

@jopnick I want to make clear that I am not advocating for the use of IDs. Quite the opposite, I don't think the client should be working with IDs at all, they should be working with URIs. But there are cases where you need to include parameters in a URI that isn't an ID. For example, queries. So you need someway to tell the client how to introduce the parameters. It is true that people could abuse URI templates, but I still think they would be useful for delineating between things that identify a resource vs. data that should be included in a payload.

[jopnick]

@xogeny gotcha. Yeah, I see your point on how these could be used for good and not for evil, and the query example is a good one.

Thinking through how to solve them w/in the current spec, I wonder if there aren't other ways to provide query params aside from templating. If the action method is GET, you have query params by default, otherwise, it seems like you could just specify those as hidden params with a POST.

Maybe if you could give the example you were mentioning of how you see query param templating. In my mind, I keep seeing problems as having task-oriented, state-building solutions. Like instead of templating zip code as a param, you'd have a select-zip action, which starts with a select state or street or something and returns the zip as a hidden param on an action. The state is built up by the server via a task-workflow instead of a client understanding how to place id's from one entity to another

edit: re-reading your example above, I missed the templated query you had. Re-framing the example you gave, it does provide how the server was going to represent this url, but it seems since you are POSTing, those would be sent in the body anyway, making the need to template in that situation a bit redundant. The other question I have about that example is: where does productCode come from? Is that an value the client got form somewhere else? wouldn't this be an example of an id which would need to be mapped?

[jopnick]

@xogeny I've been kinda tinkering with a gist working out how to do id mapping via workflows. I'm not sure if it provides a great example for the query param mapping you were describing, but this is perhaps a more sensible example of what I was trying to describe with forcing state-building by the server via workflows

[xogeny]

@jopnick Note that productCode could be a URI (retrieved from a query). I agree, you could definitely bundle all that in the POST payload without the need for query parameters and it would make much more sense because bundling a URI in a JSON payload is going to be less awkward than including it as a query parameter.

But this is part of my point. Describing payloads is very different that describing parameters in forms (which is what Siren seems to be optimized to do). Again, I think we should separate these. URI templates can be used to describe query parameters and would largely be useful for GETs. It is the POSTs, PUTs and PATCHs that I'm more concerned with. I just think Siren's current concept of fields is not adequate.

My general sense is that people a) want to keep Siren simple and b) don't have these use cases. Thats fine. But I'm still interested in this. 😃

This also brings up what I think is an additional interesting point...Siren as a payload format, i.e., POSTing Siren. It seems perfectly reasonable, to me, that you should be able to not just GET hypermedia representations from a server but also use hypermedia payloads. This allows you to do things like specify not just data in payloads, but relationships between resources. To me, this seems enormously powerful.

I think a lot of people are scared off by this because it means treating your payload as something more than JSON. But I don't really get why that is scary as long as you have good tooling. Which points to a couple of other things I've been working on. First, is to change the way web services are authored so that they don't reflect HTTP, but rather really reflect (at a fundamental level) the abstractions of hypermedia: resources, relations, actions, etc. I use something very close to Siren as an internal representation for my services. Adding HTTP is just something you can build once on top of these hypermedia abstractions. The point is that the resources should see themselves (and those resources they are related to) as resources on the server side...not functions that handle HTTP requests.

Another thing is related to how we interact with the server. One criticism of hypermedia formats in general is that they are too "chatty" (always having to fetch all these cross-linked resources). But that is, IMHO, only because we haven't evolved our thinking. In my current stuff I have created a simple Siren-based query language that is most conceptually similar to GraphQL. The idea is that a client can say "hey, here is a representation of all the information I want from you". This includes following links, filtering out fields, predicates, etc. That gets pushed to the server and the server processes all this (following links, collecting data) all on the server side and returns everything in one response. Not only less chatty, but also faster (since the traversals are all done without the overhead of lots of marshalling, unmarshallling and requesting) and pre-filtering the response to only get back what you want.

I think Siren provides a great foundation. Given that so many people seem to want to keep it simple, I suspect I'll have to simply play around with trying to find clean, orthogonal ways to extend it. I'm not sure if there is an officially prescribed way to do that, but I'll look into it.

[jopnick]

Note that productCode could be a URI (retrieved from a query).

this does feel a bit like specialized knowledge on the client: get URI from query, pick-n-place. It being a URI is great, the only part I was resisting was the client being the expert in that id mapping, or relationship instead of that being prescribed/pre-filled by the server.

But this is part of my point. Describing payloads is very different that describing parameters in forms (which is what Siren seems to be optimized to do). Again, I think we should separate these. URI templates can be used to describe query parameters and would largely be useful for GETs. It is the POSTs, PUTs and PATCHs that I'm more concerned with. I just think Siren's current concept of fields is not adequate.

That is a very interesting statement, which I'd like to hear more on. The way I've approached this was that the href in the action would have already been populated by the server in a previous step; breaking the param formatting and later, executing the action into 2 steps. This, Iinstead of doing 2 things at the same time ( templating the href as well as doing things with fields specified by the form ).

The places where I've found the siren fields inadequate was for describing complex objects, like a siren entity. It seems very concerned with 1-deep entities, and doesn't really have a language for describing a complex json object, like the entity itself.

It also appeared this was by design in some of the google group discussions, which resulted in the discussions around task-oriented apis.

To your point about hypermedia being too chatty because of this, this is a fear which keeps me up at night whilst designing. If its not secret sauce, and perhaps the topic of a separate google-group discussions, I'd also love to hear more about your GraphQL'esque query language. Seeing that used in templating might also enlighten more about how you envision it helping out

[pke]

Given a "search" rel link in the response. How would the client know how to submit the search?

[milgner]

I've just come across this issue while thinking about a way of providing clients with information about pagination: since pagination is navigational by nature, using actions doesn't really seem appropriate.

The client needs to know that there are query parameters for the page and for the number of items per page and rendering all possible permutations of these as individual link relation doesn't seem a good way to go about it, either.

Do you have any recommendations for rendering pagination information with Siren?

[mcintyre321]

@milgner a while back I put together a POC for doing paging with Siren (for .NET). I used link rels.

Example Siren API - http://apps.10printhello.com/comments

HTML browsable version: http://713fed7a6ab348379e2fecea7e7ca4d8.apinest.com/comments

Gist with the code: https://gist.github.com/mcintyre321/af68a899c30d014462e731864ae2b1a4

I'd be interested in any feedback

edit: one thing I'd particularly like feedback on is a sensible way to rels to use for sorting. I ended up coining a rel per sortable column (e.g. sortedById, sortedByName) which doesn't seem ideal.

I was thinking of maybe having a subentity per column, with a 'sortby' rel on it, but I wasn't sure if that was a misuse of Siren

[xogeny]

One simple solution would be to use actions but instead of application/x-www-form-urlencoded as the “type” use a placeholder like “rfc6570” (sadly, there is no media type for URI templates). If you think about it, this is just an alternative url encoding. Just put the template in the href field and off you go.

On Tue, Jan 29, 2019 at 5:53 AM Harry McIntyre notifications@github.com wrote:

@milgner https://github.com/milgner a while back I put together a POC for doing paging with Siren (for .NET).

Example Siren API - http://apps.10printhello.com/comments

HTML browsable version: http://713fed7a6ab348379e2fecea7e7ca4d8.apinest.com/comments

Gist with the code: https://gist.github.com/mcintyre321/af68a899c30d014462e731864ae2b1a4

I'd be interested in any feedback

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kevinswiber/siren/issues/65#issuecomment-458496233, or mute the thread https://github.com/notifications/unsubscribe-auth/ABbeuJDpoe48BviwlRt6WvGTciNNhZf7ks5vICgIgaJpZM4GU8_P .

[MathiasReichardt]

I am currently using an approach where I build all URIs at the server to have fully transparent URIs.

For queries I send a JSON containing all parameters (sorting, pagination you name it) to the server (POST Action) basically creating a Query resource. The server responds with a location header which will lead to the 1st page of the query result resource. In the real word this location URI just encodes the sent parameters to the query string. There for this resource is a virtual one (dont have a better name) which does not really exist as persisted data but is calculated from the server build URI.

The down side is the extra round trip to let the server build the location. To avoid this in some cases I add prebuild queries as link, quite similar to @mcintyre321 sorting links.