Proposal: Multisegment Path Parameters

[timburks]

According to the OpenAPI Specification version 3.0.1, Path templating refers to the usage of curly braces ({}) to mark a section of a URL path as replaceable using path parameters.

In related discussion of path parameters, text within curly braces is used as an identifier name for the matching path section. It is generally assumed but not explicitly stated that path parameters must not contain slashes (/). This is discussed in detail in Issue 892: Support for path parameters which can contain slashes.

Here we propose a change that removes ambiguity in the specification, addresses many of the needs expressed in Issue 892, and enables OpenAPI usage for a large number of APIs for which it is currently inapplicable.

The Change

We propose modifying the "path" description under Parameter Locations as follows, with new text in bold:

---

path - Used together with Path Templating, where the parameter value is actually part of the operation's URL. This does not include the host or base path of the API. For example, in /items/{itemId}, the path parameter is itemId. The text inside braces (in this case "itemId") is the path parameter name with an optional suffix modifier. Path parameter names must be valid identifiers (consisting of alphanumeric characters and underscores). If no modifier is present, the path parameter can only match a single URL path segment. If a “+” suffix modifier is present, e.g. "/items/{itemId+}", the path parameter can match zero or more URL path segments. We call these “multisegment” path parameters.

---

Discussion

This change allows slashes to be used in path parameters without escaping.

As discussed in Issue 892: Support for path parameters which can contain slashes, there is concern that this will allow ambiguous situations in which multiple API paths can match a particular request.

That concern should not prevent this change for at least two reasons:

1. Unambiguous APIs exist which use multisegment path parameters.

2. The ambiguity concern is a server implementation issue only.

Many OpenAPI descriptions are derived from handwritten implementations or other description formats. These descriptions are created to simplify client-side usage by supporting generation of documentation and client code. In other cases, teams wish to migrate their existing API definitions to OpenAPI. If an API already uses multisegment path parameters, the OpenAPI 3.0.1 Specification can not correctly describe it, making OpenAPI-based tools unusable.

Our proposed change gives client library generators and other tools the opportunity to correctly work with these existing APIs. This feature is compatible with REST APIs based on gRPC-REST transcoding system, see google.api.HttpRule and current usage in the AWS API Gateway.

Note that we do not propose or recommend using RFC 6570 for this. In RFC 6570 section 3.2.3, "Reserved Expansion" is defined as being “identical to simple string expansion except that the substituted values may also contain pct-encoded triplets and characters in the reserved set.” By allowing other characters from the RFC 6570 reserved set (defined in RFC 6570 section 1.5), reserved expansion could allow invalid URLs to be created when it is used for path template parameter values. For example, if an API user specified a value of “#one/#two/#three” for a multisegment path parameter, an RFC 6570 reserved expansion would pass this verbatim and build an invalid URL. With this proposed change, only the slashes would be unescaped - the pasted-in parameter value would be properly escaped to “%23one/%23two/%23three”.

[handrews]

@timburks I have to say I really don't understand the aversion to RFC 6570. Yes it, like nearly anything, can be used badly. But re-inventing half of it doesn't feel like a great approach.

[timburks]

@handrews I agree about wanting to use existing standards. It's certainly much easier to say "just use RFC 6570" and I would have preferred to propose that, but it has several problems:

• RFC 6570 would put a very big burden on tool makers; it is much more than 2x the complexity of this proposal. It's also a lot to explain. If RFC 6570 were incorporated into OpenAPI, I think teaching it would become a disproportionately large part of OpenAPI training.

• As described above, RFC 6570 Reserved Expansion does not escape important characters and would allow invalid URLs if we made it the standard way to handle path parameters containing slashes.

• Besides Reserved Expansion, the closest possibility that RFC 6570 offers for supporting this existing multivendor usage is to use Path Expansion with a pattern like this: {/list}. Here the "*" suffix indicates that "list" is an array of values to be joined by "/". That seems to me to be awkward syntactically and semantically for this situation and would require significant user education. (Re. syntax: people would be unlikely to consistently add both "/" and "" symbols or to understand why they should. Re. semantics: people would be unlikely to recognize that "list" must be an array and not a string.)

[handrews]

@timburks first a disclaimer: As you probably know as a contributor, you don't need to convince me of anything to get this into OpenAPI, and I'm fairly set in my opinion on this. So while I'll happily keep debating it, feel free to ignore me as soon as you're bored and/or frustrated :-) This is not something I'm trying to fight to the death, or even close.

With that said:

• The "burden on tool makers" doesn't make sense- the whole point of using existing standards is that you can therefore use existing implementations. RFC 6570 had some full implementations when I first looked at it about five years ago (I think)- I'm sure it only has more now.
• RFC 6570 has multiple levels, so OpenAPI could just declare that it uses the lower levels if complexity is a concern. But again, the point to me is that you offload the complexity to an existing implemented system that quite a few people understand. And the more systems use it, the larger that community becomes, instead of fragmenting yet another way to replace text in URIs.
• I don't see why it's OpenAPI's goal to prevent all possible illegal URIs. You can easily generate invalid URIs by typing them. That objection really feels like borrowing trouble.
• Not liking the syntax isn't that compelling to me. No engineer ever likes another engineer's syntax. Re-inventing something that has actually been successful because you object to the syntax is textbook not-invented-here-ism. [EDIT: although if you're going to re-invent something re-inventing it for UX reasons is one of the best reasons]

Anyway, as I said, you probably don't want to waste time trying to convince me. I like a lot about OpenAPI, but the project's insistence on lobotomizing (JSON Schema, although that improved dramatically in 3.0 but still outright contradicts the spec in places) or discarding (URI Templates) existing standards and re-inventing the wheel is definitely my least favorite aspect of the project.

[handrews]

(to be fair, I do realize that JSON Schema was mostly abandoned or unresponsive at the time)

[handrews]

I am sympathetic to the user education issue, I just think that's solvable, from having used URI Templates a great deal. In my experience they might be initially confusing but not a lasting problem.

[wora]

I work on Google API Platform. We have years of production experience with API specification and RFC 6570. Here is my take on it.

• API specification only needs about 2% of RFC 6570. In practice, no API developer understands the RFC 6570, not even the 2% part of it. Claiming RFC compatible adds tremendous learning curve to everyone while offers little benefit.

• Depending on URI Template library adds complexity to the application. In practice, people often just implement the 2% spec manually and it works just fine.

• As spec author, preventing human errors is our responsibility. It saves huge amount of productivity cost, similar to the billion dollar mistake with null pointers.

• The RFC 6570 is fundamentally not usable for API spec as @timburks pointed out. For API spec, the client must be able encode a path parameter into URL, and the server must be able to decode the parameter from the URL and get back the same value as the client. If some values cannot go through the process, the spec is not usable for APIs.

RFC 6570 focuses on URI generation for various use cases. For API developers, they care more about the client and the server experience. Developers only type in URL for testing purpose. For real usage, URL is used to pass data from the client to the server, that is what parameters are for. We cannot define a spec that blocks certain string values, such as "#" and "?".

We debated this topic for many months. In the end, we had to give up RFC 6570, because we could not find a way to make it work for APIs. It is much easier to define a tiny spec within OpenAPI itself, so tools developers can just read it and implement it.

[handrews]

@wora oh, I know I'm arguing a losing point here, don't mind me. Also, in all seriousness if anyone thinks my objections are derailing a more useful discussion here the admins are entirely welcome to delete my comments out of the issue or edit them down to [irrelevant opinion deleted with permission] :-D I've made my point and it doesn't need further attention.

I do appreciate your comments on your experience. It is very different than mine, which is useful to hear about.

For me, I have used level 3 URI Templates a great deal and found that they are easily understood. There's usually a "hey what's this question mark doing? / that just makes the variables query parameters" question and answer early on and then everyone goes on their way with it.

The frustration that is coming through in my comment is motivated by recently trying to apply OpenAPI to an existing API that is described by draft-04 hyper-schema. The API is not suitable for more recent, hypermedia-oriented drafts of hyper-schema so I wanted to use OpenAPI as the most widely adopted general-purpose, practical HTTP API specification system. (and there's no tooling for later hyper-schema drafts yet anyway- OpenAPI 3.x tooling may be lagging somewhat but at least it exists!)

I found it immensely frustrating because many of the JSON Schema techniques (and to a lesser degree URI Templating features) that made it easy to describe things in the API were forbidden. I'm not allowed to organize my schemas the way I want to. A lot of things are almost but not quite schemas, or are schemas plus other things that a schema could have solved, but now have to be done differently depending on whether the thing is a parameter or a body or whatever.

For someone starting from zero, I can see why it's viewed as simpler, but for someone coming from other tools, it's working with one hand tied behind my back. I eventually abandoned the effort in favor of improving the draft-04 schema tools and hoping that OpenAPI and JSON Schema converge more later.

This doesn't change my opinion that OpenAPI is the best available specification system. It is, and I don't see anyone else as even close. But it can be the best and still be frustrating.

[darrelmiller]

It is true that most people don't use most of the scenarios that 6570 support. It's also true that the vast majority of URI construction code that I see is full of broken assumptions that will fail one day on an unusual URL.

The way URL parameters were enhanced in V3 was designed to allow a subset of 6570 features, without the slightly strange syntax, while still allowing tooling folks to use existing 6570 libraries if they wanted.

In order to support the {/param*} option we would need to introduce a new style perhaps called segment and allow people to set explode to true if they want to insert multiple segments. It would be a tiny, non breaking change to the spec that is consistent with the existing approach.

The only reason this option was not added into V3 was because it introduces new challenges when matching URIs to operations. If we can address that problem, then I am fully behind adding support for multi-segment parameters. I'm not quite sure I'm ready to accept that ambiguous matching is a tool problem.

[handrews]

The way URL parameters were enhanced in V3 was designed to allow a subset of 6570 features, without the slightly strange syntax, while still allowing tooling folks to use existing 6570 libraries if they wanted.

This is a good way to compromise :-) If we could just get rid of nullable we'd have the same situation for JSON Schema.

[wora]

Existing APIs may already have ambiguity and the server handles it in a certain way that we can not change. For example, /food and /{name} are ambitious, Google APIs handle more specific match first, so /food has precedence.

The question here is do we let people express their existing APIs? OpenAPI can not force people to break their business application, likely billions of dollars a year.

[darrelmiller]

@wora Yes, you are correct, the question is definitely, should we try and support every HTTP API? From the very beginning Swagger was opinionated about the way you should describe your API. It did not try and describe every API. It become very popular with this approach. Partially because it was able to maintain a certain amount of simplicity by not supporting certain scenarios.

The question now is with OpenAPI being the de-facto choice for people who want to describe HTTP APIs, does OpenAPI inherit the responsibility to support every HTTP API. There is continual pressure to support additional scenarios, but at what point does adding more complexity reach the tipping point, where newcomers say "this is too complicated, let's invent something simpler". This has happened over and over again in the tech space.

It would be awesome if we could find a way that supports every scenario without being intimidating for folks who are new to the specification. I have said many times that I would like to add rules to the specification that would clearly define how to do URL matching. However, we have not yet had an opportunity to do this work.

It is true that there are some existing ambiguities in the way paths can be defined. And yes, most people use the same rule as Google has chosen to match literals before parameters. Because that is obvious choice and it is simple to do. However, that is the easy case, and doesn't justify introducing more ambiguity where the solution is not so obvious.

It is easy to say that we will just let the tools handle it. But then you are just pushing the problem onto all the tool makers who need to support every case. The end result is an inconsistent experience for the end user.

I look forward to hearing suggestions from folks about guidance we can provide to tool makers to do URL to operation matching for these more complex scenarios, so that we can allow more flexibility in OpenAPI descriptions.

[wora]

Thanks Darrel. Your comment makes sense. BTW, this is Hong Zhang from Google. We met recently at APIStrat.

Obviously, multisegment variables will introduce more ambiguity. This issue has been solved on the server side using different techniques. If we introduce multisegment parameters to OpenAPI, it would not solve the ambiguity problem in the same as the existing implementations on the server side. It probably is not something OpenAPI should solve.

The alternative is to introduce some standard parameter flag, e.g. explode = true, that informs the client library to expand a parameter while preserving the "/". This would solve the client-side problem while leaving the server-side problem to their existing solution. I think this is a reasonable trade-off.

One risk we should document that any client library that ignores the flag would produce incorrect URLs and cause application errors. I think this is probably better to introduce a new syntax like {name+} that would break all existing OpenAPI tools instantly.

If we want to solve the ambiguity problem, I do have some ideas. Since they are not compatible with RFC 6570, I don't think we should discuss in this issue.

Summary: I think it is very reasonable to add a parameter flag to simulate multisegment variables on the client side.

[darrelmiller]

@wora @timburks We discussed this issue at length today in the TSC meeting and there is general consensus that this problem should be addressed.

We do feel that the existing style and explode parameters are expressive enough to describe the desired scenarios if we add a new value to style. Even if we did think that adding suffixes was the better solution we could not do that without a breaking change update, as currently the characters allowed to describe parameters are not constrained.

When it comes to dealing with resolving ambiguity in URL to Operation mapping we have one proposal that we would like to get community feedback on. As a hybrid of the previously discussed possibilities, we are considering allowing tooling to deal with resolving ambiguity in whatever is the most logical way for them. However, for cases where an API designer is concerned about the precedence of matching, they could specify an explicit matching priority value in the path item. The specification would require that explicit matching priorities would overrule any other selection algorithm implemented by the tool.

I believe this approach would have the most minimal impact on both tool vendors and API designers. API Designers can ignore the concept until they run into a situation of ambiguity. Tool vendors that generate OpenAPI descriptions have the option of generating a priority value that matches their existing priority algorithm. All the spec needs is one extra optional priority property.

Your feedback would be most appreciated.

[wora]

I agree if we can enhance the existing style and explode to cover the use case asked in this issue, it is the preferred option. Many products are using the feature, as long as OpenAPI supports it, we don't need to worry about the specifics.

Having an explicit priority property does make sense. I wonder how many tools actually use OpenAPI to produce the runtime dispatching table. We should wait demand to be more obvious before we actually do it.

[jon-whit]

Any progress update on this? I need this type of support for a project I am working on.

[whitlockjc]

HttpRule supports a scenario where multi-path parameters contain static segments, but I don't see this proposal mentioning that. _(For example: path: /api/{foo/*} would make it where the foo parameter would include foo/me for an API request for /api/foo/me. Did you intend to omit this feature of HttpRule, or did we just not explain that scenario?

[handrews]

This has been rolled up into #2653, which in turn will almost certainly be moved over to the Moonwalk (OAS 4) project, which as currently proposed supports full RFC 6570 URI Templates.