Support for path parameters which can contain slashes

[dimonomid]

I see there are a few issues which ask for the opposite, e.g. this one https://github.com/swagger-api/swagger-js/pull/280 , but what I'm asking for is to add some option to the parameter, which would allow it to contain non-encoded slashes.

My use case: I have an API which allows any arbitrary path to be passed in, for example all of these:

• /api/tags
• /api/tags/foo
• /api/tags/foo/bar/baz

Are valid paths. I tried to describe it as follows:

  /tags{tag_path}:
    get:
      parameters:
        - name: tag_path
          in: path
          required: true
          type: string
          default: "/"

However, https://generator.swagger.io encodes slashes in the path, so it doesn't work. So is there a way to describe my API? It feels to me like a perfectly valid use case, so I'm surprised it's not supported.

If it's indeed not supported, then one possible solution is to add some option for the path parameter, which would allow it to contain non-encoded slashes.

What do you think?

[webron]

The request is not new, and came often around the request to support RFC 6570. At the moment, we've decided to not support it, though we'll probably re-evaluate in the future.

[dimonomid]

I see, thanks for clarifying. However, what were (are) reasons not to support it?

[webron]

As with other features, one of the things we keep in mind is tooling support, and we felt that it would complicate the requirements for OAS 3.0 compliant tools.

[darrelmiller]

@dimonomid Not allowing unescaped slashes in parameter values is for the same reason we don't allow optional path segments. Some tooling needs to be able to deterministically map a URL to an operation. With the current limitations it is easy to match the path segments in the URL to those defined in the operation because the path segments are fixed and the only ambiguity arises when a parameter value matches a constant, and IMO the constant should always win.

However, as soon as we introduce optional path segments then we have the following issue:

Given the URL path:

 /foo/bar/baz

Any one of these paths could be a match (using RFC6570 syntax for illustration only)

      "{/segments}"
      "/foo{/segments}"
      "/foo{/segments}/baz"
      "{/segments}/baz"
      "{/segments}/bar/baz"
      "/foo/bar/baz{/faz}"
      "/foo/bar/baz"

Which then requires us to introduce a priority based matching system, which can be really challenging to debug. Should we define a priority algorithm? Should the definition writer be explicit about priority? Should we use document order?

I think we do need to solve this eventually, as well as using query parameters to distinguish operations. But until we have a very well defined priority algorithm that is easy to understand, I believe we need to keep the constraints in place.

[t1]

Which then requires us to introduce a priority based matching system, which can be really challenging to debug

Isn't this a problem that the REST frameworks already do solve? OpenAPI only has to follow their lead so the documentation matches the implementation, right?

[darrelmiller]

My understanding is that some frameworks don't address it, some do, but many do it in different ways. There is no standardized algorithm for deciding on the right match. Even RFC 6570 punts on this:

Some URI Templates can be used in reverse for the purpose of variable matching: comparing the template to a fully formed URI in order to extract the variable parts from that URI and assign them to the named variables. Variable matching only works well if the template expressions are delimited by the beginning or end of the URI or by characters that cannot be part of the expansion, such as reserved characters surrounding a simple string expression. In general, regular expression languages are better suited for variable matching.

[t1]

Sure, there's no standard. And it can be a problem. But in practice I rarely encountered this to be a problem. As a developer I think about valid URIs and try to stay away from any ambiguities: They are problematic in the implementation, not in the documentation, are they?

[darrelmiller]

@t1 One of the goals we keep reminding ourselves of when updating this spec is we want it to be as straightforward as possible for people to build tools around it. If we define behaviour but leave ambiguities then we could end up with a ecosystem of tools that are not interoperable. That would not be good. Yes, you an individual usage of OpenAPI can avoid these ambiguities, but ensuring everyone avoids them in the same way is not nearly as easy. We do want to achieve this, we just couldn't fit it into the 3.0 timeline. I can assure you that when we designed the current URI template support, we made sure adding the / prefix would be an easy non breaking change. We will simply introduce a new style value.

[wohnemus]

I believe this misses an important point regarding REST APIs, however. It is critical, in fact, that we make our REST APIs discoverable. What does this mean? It means that our Paths are going to be variable and discovered at run-time. Using hypertext (HATEOAS principle) combined with a properly constructed OPTIONS response should make a REST API truly discoverable. Prescribing a specific path definition in the specification actually precludes this sort of dynamic discovery. Having a variable path enables this sort of discovery. It does mean that debugging may be more difficult. It also means that current tools may not be able to understand the specification, but both of those are implementation requirements not design requirements. OpenAPI, I presumed, was a design specification and that as such should be tailored to the needs of REST API design including fully Level 3 compliant (Richardson maturity model) REST APIs. Insisting on a fixed path like this does seem to go against these concepts. APIs we are developing are intended to be RMM level 3 compliant and discoverable. We also use OpenAPI and this creates a conflict for us. We'd hoped that in version 3 this would have been resolved.

[darrelmiller]

@wohnemus OpenAPI has never had the goal of describing a dynamic hypermedia API. The idea of a static API description and hypermedia are contradictory goals. I spent 6 years building a hypermedia based system and had no need for something like OpenAPI.

Most people building what they describe as a REST API are not trying to build a hypermedia API. They are building a HTTP API, which OpenAPI does a good job of describing. OpenAPI v3 introduces the notion of linking, but it is a design time linking concept, not a runtime one.

There are a number of people in the OpenAPI community who would like to see more support for hypermedia style APIs. We may be able to do something to help that in a future release, but as I've said before, the last thing I want is to have OpenAPI do hypermedia badly. And considering OpenAPI is all about communicating to a client an exact description of how a HTTP API works, and hypermedia is all about following your nose, reconciling those two successfully is not going to be easy.

If you have ideas as to how client/server coupling can be kept at a minimum whilst still using OpenAPI, I'd love to hear it. If we can do it successfully, then I will be fully behind it.

[handrews]

@wohnemus if you want a fully dynamic HATEOAS-driven REST API, I encourage you to come over to the JSON Schema project and help us out with JSON Hyper-Schema, which explicitly has those goals as opposed to the static description goals of a service definition like OpenAPI. The two projects serve different purposes (which @darrelmiller nailed perfectly in his hypermedia API vs HTTP API distinction) and I think Hyper-Schema may be more suited to your needs. We are doing a lot of work on JSON Hyper-Schema right now.

I hope that eventually as JSON Hyper-Schema matures, OpenAPI will find ways to adopt it, but Hyper-Schema is definitely not there yet. We're still working out the best practices and features for hypermedia APIs, and it's probably best that we do that separate from any larger project targeting static description such as OpenAPI.

[wohnemus]

@darrelmiller I understand the goals of OpenAPI but it is also true that vendors like WSO2, IBM and many others are adopting OpenAPI as the standard for their tools. As an API developer I need to use such tools for REST APIs, including those that are dynamic. This creates that conflict. @handrews we'd love to join in the discussion there too. We are making use of the HAL-JSON specification for this purpose. The schema definition and design married with OpenAPI specification will help support dynamic APIs.

[handrews]

@wohnemus and anyone else interested in hypermedia description: we're in the final review phase for publishing Draft 06 for the next (probably) 2 weeks or so, but after that I hope to dig into Hyper-Schema a lot for Draft 07. Our main limitation with Hyper-Schema is that we have very few people who are really looking to use it in actual HATEOAS-driven APIs, so getting more actual potential users in the discussion would help us move it forward.

[t1]

@darrelmiller: Maybe the spec should just ban ambiguities, so tools will mark them as errors. If somebody chooses to ignore that, some implementation may still work due to priority management... but that's completely outside of the standard (and utterly confusing).

[webron]

Everyone, just so we're on the same page - the @OAI/tdc has decided this will not be supported in 3.0. This is not to say the requirements and needs are not clear, but we can't afford to reopen it for discussion. It's time for us to make the cut for what gets in and what not.

We encourage further discussions so we can see if/how this gets in future versions.

[bassmanitram]

@handrews You'll be glad to know that Bill ( @wohnemus ) and I (and our teams), are heavily using JSON Schema and JSON Hyperschema in our API definitions and documentation - including in the dynamic description of what the API can do. We'll be coming over with great alacrity :)

@darrelmiller Exactly the approach I was about suggest. AWS API Gateway uses OpenAPI for its API descriptions and simply allows you specify an API to a certain depth and then have a path parameter described as

/seg1/seg2/{restOfPath+}

This basically says ... well it's obvious: "swallow the rest of the path". Yes, I know this isn't standard OpenAPI nor is it standard OpenAPI Extension syntax - but they saw the need and did it. It avoids ambiguity because, after that, you aren't allowed to create ambiguous paths.

IBM's API Connect also uses OpenAPI - but has not chosen to allow this feature.

@darrelmiller Darrel, first thanks for your efforts - a thankless task! I completely get your point that the level of dynamism we're going for doesn't require OpenAPI per sé, but, as the two product citations above imply, in order to get our APIs published and marketed (maybe that's a dirty concept here - sorry :)) we need to put them in a "standard" gateway and the developer portal facilities that those gateways may provide - and that usually means conforming to the best declarative standard out there - OpenAPI. So we end up being constrained by the number of extensions (and liberties) that the implementors add to OpenAPI.

Obviously AWS has taken liberties to get it to work, whereas IBM has constrained itself to the OpenAPI standard in this case, meaning we have to work around it (figure out your longest path, then describe a path parameters for each segment in that path - yuk).

So (and admittedly not having thought this through to the extent I know you'll be able to) would something "restrictive" like AWS's solution be possible as a first pass in, say, version 3.1?

[darrelmiller]

@bassmanitram 6570 URI Templates support what you describe,

/seg1/seg2{/restofPath}

We can easily add this syntax by introducing a new style value. That's not a problem.

I do understand the need. I work on Azure API Management and we support templates like /seg1/seg2/* that does the same thing, and we also use OpenAPI to describe our APIs. So, we have the challenge of OpenAPI not being able to describe some APIs that our customers are defining.

Only using wildcards at the end of a path is one way to limit the ambiguities and it does cover a fairly significant number of use cases. However, it would be nice to find a way to address this in a more general way so that we can solve it for query parameter discrimination too.

Once we get 3.0 out of the way, you can be sure this issue will be high on my list of priorities.

[bassmanitram]

@darrelmiller Re RFC 6570: Exactly. I mentioned AWS's approach simply as an example of a provider who facilitates the functionality in their product. Their syntax is clearly not 6570-compliant, more regex-inspired I suspect.

All understood (Azure API Management is in two weeks time for me - thanks for the heads up :)).

Thanks

[handrews]

@bassmanitram great! Please feel free to introduce yourself on our quiet but still functional mailing list or grab my email address out of the the current drafts on master and say hi :-)

[dimonomid]

So this is not going to be supported soon, and I have to resort to a workaround.

If I have a path /tags{tag_path} and I enter /foo/bar as tag_path, then the actual query request URL will be: /tags%2Ffoo%2Fbar. So, I just added support for that on my backend: the endpoint handler for /tags* urldecodes the path (which is %2Ffoo%2Fbar), and it becomes /foo/bar again.

Yes, a hack, but it works, and it's better than nothing. In my case, tag names can't contain the / character, so there's no conflict. Your mileage may vary, of course.

[rsleggett]

We took a different approach and changed our API to use query parameters instead.

[ePaul]

@wohnemus I described a way of having hypermedia support with an extended variant of OpenAPI in #577. I just need to get to implementing it ...

[bassmanitram]

@ePaul WE ( @wohnemus me and others from our org) could have written you first couple of posts in that issue - it's spot on for what we too are trying to achieve. Now need time to digest it and start commenting on it - first glance looks very promising.

[EtienneK]

+1

[timburks]

I think opposition to this is resulting from a particular way of thinking about API descriptions.

Many OpenAPI descriptions are what I'll call derivative: they are generated from service implementations or other API description formats. These descriptions are not the primary source of information about an API but they are still used to drive tooling. Usually this is on the client side, in particular for producing documentation and client libraries. For these cases, supporting slashes in path template variable bindings adds no ambiguity, because users know what methods they want to call when they call functions in a client library. But explicitly acknowledging the possibility of slashes does allow API client libraries to correctly generate API paths when these slashes should be preserved and not escaped.

Many of us have a tendency to think of OpenAPI descriptions in a stronger, definitive sense. In these cases, OpenAPI descriptions are the primary representations of services and service code is generated from them. This is a great use of OpenAPI that I myself advocate, but I don't think it is appropriate to limit the expressiveness of derivative descriptions just so that definitive descriptions are implementable. (The relevant problem is that slashes in template variables can lead to ambiguous server-side path matching).

Instead, we should use linters to make sure that descriptions are implementable (when we need them to be) and be liberal in what we accept as valid OpenAPI.

[timburks]

Here is a specific proposal to address this. It supports current AWS API Gateway usage and a large class of Google APIs that are currently unsupportable by OpenAPI.

[jon-whit]

What's the status on this? I actually just came across this need for HTTP --> gRPC transcoding to work properly.

[bleuscyther]

This is what i have from swagger : https://swagger.io/docs/specification/paths-and-operations/

[nickfaughey]

My 2 cents for anyone sent here by Google - if you have control over the client, it's going to be a much easier short-term solution to drop the variable length parameter in a query param instead: /api/tags?path=foo/bar/baz

/tags:
    get:
      parameters:
        - name: 'path'
          in: query
          schema:
            type: string

Depending on your implementation, path might need some url decoding. A workaround for sure, but just a reminder to check if your use case allows you to tweak the client URLs a little, because that'll be a much faster implementation than waiting for OAS v4+.

[patrickelectric]

Hey, is there any update related to it ? Or any possible deadline/version release with this feature ?

[garymazz]

I'm also interested in this feature, without the curly braces. I'm attempting to place an international storage standard in swagger, however the path implementation does not correspond to RFC 6570. In our case, the last segment of the path refers to the object without a preceding path discriminator symbol or segment tag.

Can we use a path and parse it ? Yes, however it breaks the documentation which is one of our priorities for selecting swagger and OAS.

IMO, if AWS is providing this feature, it means many of their customers are using it. AWS has a tendency to find the lowest mark to support. If they're supporting a feature, it must be in great demand.

Is the OAS team attempting to force a change to the user community ? Or improve adoption of OAS/swagger ? Or let the user community develop another specification to meet their needs ?

[erizzo]

I do understand the need. I work on Azure API Management and we support templates like /seg1/seg2/* that does the same thing, and we also use OpenAPI to describe our APIs. So, we have the challenge of OpenAPI not being able to describe some APIs that our customers are defining.

@darrelmiller I know it's an old message, but is that support for "wildcard" documented anywhere in the APIM documentation? I want to propose it as an option to some Powers That Be but they may ask me to demonstrate that it's officially supported.

[segevfiner]

This is very commonly supported in various web frameworks. For example:

1. Pyhton's Django & Flask have a path convertor, which accepts a path parameter with slashes (Basically the rest of the URL).
2. gin & echo have an * syntax in their routes for this.
3. express.js support an asterisk to denote a zero or more parameters.

It was even used in Gitea's API for some file related APIs, but to wrong effect in Swagger/OpenAPI as they can't specify that the parameter is unescaped, so it is broken in swagger-ui's "Try it out" or in generated clients, etc.

[handrews]

@darrelmiller @webron reading back over that quote from RFC 6570:

Some URI Templates can be used in reverse for the purpose of variable matching: comparing the template to a fully formed URI in order to extract the variable parts from that URI and assign them to the named variables. Variable matching only works well if the template expressions are delimited by the beginning or end of the URI or by characters that cannot be part of the expansion, such as reserved characters surrounding a simple string expression. In general, regular expression languages are better suited for variable matching.

The solution to disambiguating could be quite simple: allow specifying a regex as part of the Path Object for the reverse match. By default, OAS would do the best effort template reverse match, which in most cases would work. But for a "full RFC 6570" style or a "pathExpansion" style or something like that, the regex (if present) could be used instead.

(I'm not advocating this on any particular time scale, it just occurred to me reading over the issue so I thought I'd make a note of it).

[garymazz]

@darrelmiller @webron I'm in agreement with @handrews. Reference scanning should be part of disambiguation. Support of regex substitution as a preconditioning stage would translate incompatible syntaxes to match swagger/OAS capabilities. Although a dirty workaround, it would provide a way to align paths currently incompatible swagger/OAS capabilities. To get cloud standards including CDMI supported by swagger tools, I placed regex substitution prior to path dispatch matching.

[handrews]

@garymazz I'm pretty sure anything involving regexes is a dirty workaround, including the entire Perl programming language 😜

[garymazz]

@handrews lol, yeah. Its ok in java and c/c++. The performance of javascript a bit slow, but should be written and executed as optimized native code. In javascript I just injected the translator, in c++, I hacked the libraries. I can't complain about perl (or tcl), I made a good living with it back in the day.

Another work around is making code generators support hooks. Just an observation, the translator is really just a reverse proxy...

Enough talk about workarounds, I would like to see OAS syntax extended or at a minimum formalize predispatch substitutions.

[rcollina]

Hello, I recently needed this as well.

/resources/{resourceRef+}

And it would be extremely helpful if it matched the following, and exposed metadata accordingly:

/resources/123 /resources/123/action /resources/123/subresources/456

Is there any plan to support this?

[jypma]

Could someone from the OpenAPI core team leave a final comment on the likelihood of this being added to a future spec? The most common cases I see summarized here are:

1. Allow a * to match a path and any subpath after it : /myapi/tags/animals/mammals/tiger
2. Allow a path parameter to be repeated : /myapi/tags/animals/mammals/tiger/tag_properties

The argument against this feature was to be able to automatically match requests against documentation, which of course should be retained. But both (1) and (2) do not need to cause problems, if some simple rules are set up, e.g.

• A * at the end of a path is only tried when less-specific paths don't match
• Only one repeated path parameter is allowed in a path
• Specific paths are always matched before a repeated path parameters when matching a path

The rules could even be more strict than that.

[royteeuwen]

+1. I use Adobe Experience Manager / Apache Sling at our customer and they work based paths that have specific selectors / extensions. So we would like to use the {path}.{selector}.{extension} where path is not slash encoded.

[momegahed]

It would still be nice to have this feature

[duncant-wssu]

Does anyone happen to have a short C# snippet to allow the backslash within Power Automate Custom Code in the path?

[mzihlmann]

the whole point of openAPI specification is to describe existing apis, not to prescribe which apis are "good" and which are not. The ambiguity is not on the client, not on the documentation, and therefore not on openAPIs side, but purely on the server. The webserver frameworks already solve this ambiguity, they all provide sane possibilities to have paths in urls, openAPI just has to its job and describe these api endpoints.

i'm sorry for the rant, i really don't want to hurt anyone, i think openAPI is a very good project, but pointless implementation gaps like this can be really frustrating.

[mzihlmann]

just fyi, i ended up working around this limitation not in the api implementation itself, but by simply inserting a nginx-proxy between swagger and the api (this has no effect on the productive environment in my case as swagger is only run locally, not on the server). The proxy is responsible for url-decoding the request, this way the correct path-like parameters are transmitted to the api on the server. Only downside is that in the swagger-UI the request is still shown wrong.

server {
    listen 80;

    location ~ /api {
        # swagger cannot handle '/' in url parameters
        # there is no option to not url-encode them.
        # https://github.com/OAI/OpenAPI-Specification/issues/892
        # Therefore, instead of decoding on the target
        # we decode them here in the reverse proxy.
        # nginx $request_uri is the original,
        # $uri is the url-decoded version.
        proxy_pass http://${IP_DUT}$uri$is_args$args;
    }
}

[t1]

just fyi, i ended up working around this limitation by moving to GraphQL ;-)

[pr-ebsco]

I also have a similar problem, Trying to get entity using entityId which is a URL.

API: GET /v1/entities/{entityId} Example: GET /v1/entities/https%3A%2F%2Fwww.google.com%2F This is throwing 404 but working with a simple string instead of a URL

api-spec.yaml

'/v1/entities/{entityId}':
    x-eis-wip: true
    get:
      parameters:
        - name: entityId
          in: path
          required: true
          schema:
            type: string

Tried /v1/entities{/entityId}, /v1/entities/{entityId+}, these are throwing validation error at gradle openApiValidate

Please help!

[developedsoftware]

just fyi, i ended up working around this limitation by moving to GraphQL ;-)

By far the best comment on this thread !

[pkit]

I'm in awe that it's 2022 and there is no way to describe any object storage API in an OpenAPI "standard"

just fyi, i ended up working around this limitation by moving to GraphQL ;-)

I'm not sure how GraphQL handles url path, last time I've checked - it doesn't.

[darrelmiller]

@mzihlmann

the whole point of openAPI specification is to describe existing apis, not to prescribe which apis are "good" and which are not.

Actually, that was not the original goal of OpenAPI. When Swagger was introduced it was intentionally opinionated about how to build APIs in order to reduce the complexity of the description. Over time we have added support for more API shapes with the ultimate goal of being able to describe any HTTP API. However, every time we add support for another niche design pattern we introduce complexity that affects every tooling author and potentially users.

There is OpenAPI tooling that works on the server at runtime, that is dependent on being able to unambiguously identify operations from URLs.

I have been advocating for a simple priority attribute to address the ambiguity for many years now, but so far, I have failed to make a sufficiently convincing argument for it. You are welcome to join our weekly calls to make a case for adding this capability. This is a community driven project. Everyone is welcome to contribute.

[darrelmiller]

In case you have a strong need to describe multi-segment paths and you don't care about ambiguous URLs, then I believe this is how you could describe it in a way that would be compatible with how it would be implemented if we were to add it to the spec. I used an x- extension to represent it. If you use this and can convince tooling authors to adopt it then it will be much easier for us to add it to the spec.

name: folderPath
in: path
description: multi-segment folder path 
required: true
schema:
  type: array
  items:
    type: string
style: simple
x-multiSegment: true