Support interdependencies between query parameters

[cwarny]

It would be great to be able to specify interdependencies between query parameters. In my app, some query parameters become "required" only when some other query parameter is present. And when conditionally required parameters are missing when the conditions are met, the API fails. Of course I can have the API reply back that some required parameter is missing, but it would be great to have that built into Swagger.

[AML14]

So this is an open issue since January 2015. It has received 50 comments from more than 30 participants in the last 5 years, so there's clear interest in it. What's more, inter-parameter dependencies are extremely common in RESTful APIs. That was our initial thought in my research group and we have confirmed it by carrying out a thorough study on 40 real-world industrial APIs (~2600 operations), including those from YouTube, PayPal or Github itself, for instance.

We have recently published a paper regarding this issue at the International Conference on Service-Oriented Computing (ICSOC). A preprint is available here. We have also released the resulting dataset, which is available here. We really hope these resources can provide a better understanding of the recurrence and dimension of this problem.

To summarize our findings, 85% of the APIs studied have inter-parameter dependencies. They are present in all kinds of operations (GET, POST, etc.), even in those with very few parameters. Dependencies usually involve 2 query parameters, but we found uncommon cases like dependencies involving 10 parameters or involving parameters of different types (e.g. body and header parameters). All in all, we identified seven patterns of dependencies. All this is explained in the paper in detail.

We are currently working on an OAS extension for the expression of these seven patterns of dependencies, including, of course, all examples given by the participants in this thread. In our opinion, doing a revision of how dependencies emerge in practice was an essential step to tackle this problem. Now that we know what we are facing, we hope it'll be much easier to propose a solution.

[reuzel]

Why not have required support lists of parameter combinations of which oneOf/anyOf/allOf must be present? Required would implement allOf implicitly for backwards compatibility. If you can nest these, most of the combinations described in the article of @AML14 can be supported. It should be not too hard to create validation logic for this.

 required:
    - alwaysPresentParam
    - oneOf:
        - mutuallyExclusiveParam1
        - mutuallyExclusiveParam2
    - oneOf:
        - allOf:
          - dependentParam1
          - dependentParam2
        - allOf:
          - dependentParam3
          - dependentParam4

[AML14]

Why not have required support lists of parameter combinations of which oneOf/anyOf/allOf must be present? Required would implement allOf implicitly for backwards compatibility. If you can nest these, most of the combinations described in the article of @AML14 can be supported. It should be not too hard to create validation logic for this.

 required:
    - alwaysPresentParam
    - oneOf:
        - mutuallyExclusiveParam1
        - mutuallyExclusiveParam2
    - oneOf:
        - allOf:
          - dependentParam1
          - dependentParam2
        - allOf:
          - dependentParam3
          - dependentParam4

I think it's more complicated than that, @reuzel. What about the parameters that forbid the use of other parameters? Take for instance the relatedToVideoId parameter in the search operation of the YouTube API:

If relatedToVideoId was combined with any parameter other than those eight listed there (e.g. videoDefinition), the API would return an error.

This is just one example, but there are many more use cases that need to be comtemplated. My team and I are working on it.

[MikeRalphson]

Glad to see this work and look forward to the proposals. I personally think it is unlikely OAS would adopt JSON Schema applicator keywords such as allOf, oneOf, anyOf or not (or properties with the same names) outside of schemaObjects, in order to avoid confusion.

[handrews]

I just proposed a correlation approach in OAI/OpenAPI-Specification#1635. Here it is:

In OAS 3.1 we could solve this by adding a new OAS-specific schema extension keyword that could be used to correlate across parameters, headers, and bodies across requests and responses. You would use a oneOf in the schema of each thing that needs to correlate, possibly with an if/then in each branch if you want to explicitly call out the predicate for the server to test.

The extension keyword would be used in each oneOf branch to name it. Since that keyword would indicate that you are correlating things, an if/then alongside of the keyword would be taken as expressing the correlation predicate (there are other reasons you might use an if/then so it's useful to disambiguate the desired behavior). You wouldn't necessarily need to use if/then in your oneOf branches, but some people find it more clear.

If you want to forbid certain parameters or headers in a particular branch, you can set their schemas to false (which is the same as "not": {} but with clearer intent).

That's all pretty hand-wavey. I can work up an example if there is interest. But this basically lets you act like there's a big oneOf over the whole Operation Object, but without actually having to support JSON Schema applicator keywords outside of the Schema Object.

Of course we could also solve this by restructuring the Path Item or Operation Objects, but that would have to wait until OAS 4.

[ASHISHTOM]

I just proposed a correlation approach in OAI/OpenAPI-Specification#1635. Here it is:

In OAS 3.1 we could solve this by adding a new OAS-specific schema extension keyword that could be used to correlate across parameters, headers, and bodies across requests and responses. You would use a oneOf in the schema of each thing that needs to correlate, possibly with an if/then in each branch if you want to explicitly call out the predicate for the server to test.

The extension keyword would be used in each oneOf branch to name it. Since that keyword would indicate that you are correlating things, an if/then alongside of the keyword would be taken as expressing the correlation predicate (there are other reasons you might use an if/then so it's useful to disambiguate the desired behavior). You wouldn't necessarily need to use if/then in your oneOf branches, but some people find it more clear.

If you want to forbid certain parameters or headers in a particular branch, you can set their schemas to false (which is the same as "not": {} but with clearer intent).

That's all pretty hand-wavey. I can work up an example if there is interest. But this basically lets you act like there's a big oneOf over the whole Operation Object, but without actually having to support JSON Schema applicator keywords outside of the Schema Object.

Of course we could also solve this by restructuring the Path Item or Operation Objects, but that would have to wait until OAS 4.

I just proposed a correlation approach in OAI/OpenAPI-Specification#1635. Here it is:

In OAS 3.1 we could solve this by adding a new OAS-specific schema extension keyword that could be used to correlate across parameters, headers, and bodies across requests and responses. You would use a oneOf in the schema of each thing that needs to correlate, possibly with an if/then in each branch if you want to explicitly call out the predicate for the server to test.

The extension keyword would be used in each oneOf branch to name it. Since that keyword would indicate that you are correlating things, an if/then alongside of the keyword would be taken as expressing the correlation predicate (there are other reasons you might use an if/then so it's useful to disambiguate the desired behavior). You wouldn't necessarily need to use if/then in your oneOf branches, but some people find it more clear.

If you want to forbid certain parameters or headers in a particular branch, you can set their schemas to false (which is the same as "not": {} but with clearer intent).

That's all pretty hand-wavey. I can work up an example if there is interest. But this basically lets you act like there's a big oneOf over the whole Operation Object, but without actually having to support JSON Schema applicator keywords outside of the Schema Object.

Of course we could also solve this by restructuring the Path Item or Operation Objects, but that would have to wait until OAS 4.

when will OAS 4 will release??

[grexe]

back-linking relevant SO thread with some proposed workarounds for OAS 3.0: https://stackoverflow.com/a/49199240/160799

[thealjey]

The only clean solution imo is to not have parameters defined as an array type, but instead make it work the same as requestBody does, then one would be free to use oneOf or anything they like. On top of that, parsing and validating of such a shape would be easier, since you can just reuse the existing requiestBody logic. But, then the in field would make no sense, and we would need to have separate keys for GET as well as PATH parameters, which if you ask me, would be a big improvement.

[AML14]

Hi everyone. I'm happy to let you know that my team and I have finally made public our proposal for handling inter-parameter dependencies in OAS. It is a domain-specific language (DSL) called IDL (Inter-parameter Dependency Language), which supports the seven patterns of dependencies that I referred to in my previous comment in this thread. We have also implemented a Java library for performing a number of analysis operations in these dependencies, e.g., to check if a given API request satisfies all dependencies.

It is all summed up in these two Medium posts:

1. Inter-parameter dependencies in REST APIs (5-min read).
2. Handling inter-parameter dependencies in REST APIs with IDL4OAS (7-min read).

While the second post describes the DSL and the analysis tool, the first one can also be useful for alternative implementations of both resources, since it describes the most common types of dependencies appearing in real-world APIs.

All this work is based on a paper recently accepted in the journal IEEE Transactions on Services Computing (preprint available here).

Hope you enjoy it!

[MikeRalphson]

@AML14 congratulations on publishing your research findings and your proposal. Is there a BNF grammar for the DSL available? One thing which occurred to me is: can IDL distinguish between two parameters with identical names which only differ by their in: field - i.e. two parameters called clientId one in: query and one in: header, one of which must be present?

[AML14]

Thanks @MikeRalphson. You can have a look at the IDL grammar here. We also provide a simplified version of the grammar in the paper, attached at the end of this comment.

Regarding your question, it's a good point. Currently no, but just because it's not a common scenario. Since IDL is implemented with Xtext, it can be easily extended. For example, it could be expressed as follows:

OnlyOne([q]clientId, [h]clientId);

[q] and [h] would be optional elements, used when this information needs to be specified. The IDL grammar can be updated to support this syntax, and the IDL parser and code generator could thereby handle such a scenario.

[AML14]

I just opened an issue in the OpenAPI Generator repository: https://github.com/OpenAPITools/openapi-generator/issues/8722

It is a feature request, which we are willing to start developing, but would like to get as much feedback from the community as possible. The point is: Based on the IDL dependencies included in OAS, we could automatically generate built-in assertions in the source code of clients and servers (any programming language) that handle the validation of such dependencies.

I think this was one of the main ideas when this issue was opened, but it's going to be big, so we need to discuss it all together and provide our own views on the matter.

[SteveNay-vz]

This issue is now over 6 years old. :-( OAS 3.1 was just released yesterday. I don't want to do any disservice to the IDL solution offered by @AML14, but was anything done to support inter-parameter dependencies in v3.1?

I'm not overly familiar with JSON schema (now fully supported in 3.1), but I spent a little time reading about dependencies and "applying subschemas conditionally" (if/then/else), and I don't see how they can help with inter-parameter dependencies. Unless I missed something, both of those concepts only apply within an object, but query parameters are completely separate objects, each with their own schema.

If JSON schema can be used to describe relationships between query parameters, I would really appreciate it if someone can point me to where that is described and perhaps an example. Or, if IDL4OAS is the officially recommended method of describing parameter relationships, let's just say so and close this issue and then all go have a drink to celebrate.

[MikeRalphson]

@SteveNay-vz

I don't want to do any disservice to the IDL solution offered by @AML14, but was anything done to support inter-parameter dependencies in v3.1?

No, it wasn't a focus during development of v3.1. That development started in October 2018, I don't think any non-breaking proposals were made in that time-frame, except adding a specification-extension, which of course, anyone is free to do at any time. We're always interested in people's experiences with feature pilots.

I'm not overly familiar with JSON schema (now fully supported in 3.1), but I spent a little time reading about dependencies and "applying subschemas conditionally" (if/then/else), and I don't see how they can help with inter-parameter dependencies. Unless I missed something, both of those concepts only apply within an object, but query parameters are completely separate objects, each with their own schema.

You haven't missed anything. Several people above have suggested using a single JSON Schema object to represent query parameters, path parameters and headers. That poses some problems (do you have three objects or one, how are style and explode handled - as a new JSON Schema vocabulary?) but I suspect it will certainly be discussed for v4.

Or, if IDL4OAS is the officially recommended method of describing parameter relationships, let's just say so and close this issue and then all go have a drink to celebrate.

IDL4OAS was only proposed during the last month of v3.1 development, when we were already into the RC phase. We haven't yet seen enough real-world feedback on it to even evaluate it properly, let alone recommend it. Darn, no drink for me. 😿

This issue is now over 6 years old. :-( OAS 3.1 was just released yesterday.

Unless GitHub have removed helpful features, this is a little redundant.

[AML14]

IDL4OAS is only as powerful as we, as a community, want it to be(come). Thanks to it, we've been able to develop tools that leverage the inter-parameter dependencies in the API, for example:

1. IDLReasoner: a tool for performing multiple analysis operations such as checking if the dependencies are satisfied in a request. So, for instance, it could be deployed on an API gateway to reject all requests that violate dependencies, without redirecting those to the respective service.
2. RESTest: a testing framework for REST APIs that generates requests satisfying/violating the inter-parameter dependencies of the web API operation. Internally, it uses IDLReasoner.

I just announced the opening of that issue in openapi-generator because I think automatic code generation is one of the greatest strengths of OpenAPI. If we can automatically generate the code in charge of checking if inter-parameter dependencies are satisfied or not in API clients and server stubs, that will be awesome.

[AML14]

For those interested in automatically generating built-in assertions in the source code of clients and servers, based on the IDL dependencies present in the OAS document (using our IDL4OAS extension), we just implemented a preliminary feature in OpenAPI Generator: https://github.com/OpenAPITools/openapi-generator/issues/8722.

It currently targets Java (several frameworks of clients and servers), but it could be easily extended to other languages and frameworks in the future, especially taking @enriquebarba97's implementation as a reference.

All feedback welcome. Hope this is the beginning of some new cool stuff for the OAS ecosystem 🥳.

[vellala2000]

issues like these are critical for Open API adoption. Hope we will have solution for this in next version of openapi release.

[jon-jon-jon]

Find my solution here https://github.com/OAI/OpenAPI-Specification/issues/182#issuecomment-961767168

[kaiserbergin]

Would be really rad to have some movement on this as it's fairly common to have interdependencies among parameters. Translating and interpreting what's required in those scenarios is a bit difficult in the current state, outside of comments in descriptions.

[karenetheridge]

This can be done in the JSON Schema, if all the query parameters are treated as one "parameter" and passed to the schema evaluator as an object type.

The parameter deserialization process is not specified very clearly, but I do something similar when handling header parameters -- if the "type" keyword is present at the top level of the json schema for the header, and it is "array", then multiple values of that header are permitted and they are passed to the evaluator as an array. If it's "string", then only one header value is allowed; same thing for "number" or "integer", but in that case the parsed header value is also converted to a number (if possible) before being sent to the evaluator.

[handrews]

This will be addressed by the OAS 4 Moonwalk project. Active discussions include:

• https://github.com/OAI/moonwalk/discussions/20
• https://github.com/OAI/moonwalk/discussions/24
• https://github.com/OAI/moonwalk/discussions/31

As this change cannot be made in the 3.x line, I am closing it in favor of the Moonwalk discussions.