discussion: Semantic validation checklist

[whitlockjc]

I think it would be great to keep a running list of all "semantic validation" (validation not performed via JSON Schema either due to bugs, like 2.0's array items, or due to runtime processing) As I rewrite sway for 3.x support, I've come up with the following validations (of the OpenAPI Specification Document itself, not runtime processing like readOnly). Below is this list and I would like to verify my findings, fill in any gaps and eventually turn this into it's own documentation file for tooling authors.

Semantic Validation

Semantic validation is performing validations on top of the structural validation provided by JSON Schema.

Common

Description	Type
Operations must have unique (name + in combination) parameters	Error
Operations must have unique operationId	Error
Parameters must be unique (in + name combination) at their definition path (path-scoped parameters can be overridden at the operation-scope)	Error
Path parameters declared in the path string need matching parameter definitions (Either at the path-level or the operation)	Error
Path parameter declarations do not allow empty names (/path/{} is not valid)	Error
Path parameters definition (Either at the path-level or the operation) need matching parameter declarations	Error
Path strings must be (equivalently) different (Example: /pet/{petId} and /pet/{petId2} are equivalently the same and would generate an error)	Error
Referenceable definitions should be used by being referenced in the appropriate way	Warning
References must point to existing documents or document fragments	Error
Tag names must be unique	Error
The default property for Schema Objects, or schema-like objects (non-body parameters), must validate against the respective JSON Schema	Error
The discriminator property for Schema Objects must be a required field	Error
The items property (where available) must be an Object (Unlike JSON Schema which Array or an Object)	Error
The name used in Security Requirements must match a Security Definition/Schema name	Error
The required properties for a Schema Object must be defined in the object or one of its ancestors	Error

OpenAPI 3.0.x

Description	Type
All names for #/components/* must match the following regex ^[a-zA-Z0-9\.\-_]+$	Error
Runtime expressions of Link Objects using $request.{parameter} must have matching parameter defined on its parent	Error
Server variables declared in the server object's URL need matching variable definitions	Error
The name used for Media Type Objects must have a matching properties entry for the schema	Error
The operationId property of the Link Objects must have a matching Operation Object	Error
The operationRef property of Link Objects must point to an Operation Object	Error

Swagger 2.0

Description	Type
Operations cannot have both a body parameter and a formData parameter	Error
Operations must have only one body parameter	Error
Parameter constraint properties should match the parameter type	Warning
Parameters should not have a default property when required is true	Warning
Parameters with type of file must have a consumes of multipart/form-data or application/x-www-form-urlencoded	Error
Parameters with type of array must have an items property	Error

[kylef]

In Apiary and our surrounding tooling and OpenAPI parsers/validators, we are trying to follow the robustness principle in the sense that we lean towards warnings over errors wherever possible. We will not emit an error unless absolutely necessary. For us the semantics for error is that an error would cause our parsers and tooling to exit and not continue. Which is similar to most compilers (clang, gcc etc).

For example, in Apiary you cannot save an API Description Document that contains any errors. Thus, unless absolutely necessary (such as the parser cannot continue) we will lean towards a warning. Here's some examples:

• If validation of an optional property fails we can treat that as a warning and discard the property. Such as if the required property in the 'Parameter Object' is set to a non-boolean value.
• For path type 'Parameter Object's, the required property is required and must be set to true. Our parser will warn if the user does set this value correctly. We can continue and always interpret the parameter to be required.
• The required paths property of the 'OpenAPI Object' is missing would cause a warning and the behaviour is equivalent of paths being set to an empty object.

We've seen that some of our users who are using OpenAPI 2 generators that end up with semantically invalid specifications which hinder the user's ability to use them in Apiary if we was to outright treat every validation failure as an error. We attempt to be lenient on the user where possible so they can continue to use their API Description Documents in Apiary. We've found cases in the past where users will discover that their (supposedly valid OpenAPI 2.0 document) could be used in other tooling but could not be used in Apiary due to more strict validation which prevented the user using Apiary without altering their (possibly very large and generated) OpenAPI specification.

Another reason behind this behaviour in our stack comes from API Blueprint, we attempt to never break compatibility with a document that has previously been parsable in Apiary. Thus we rarely attempt to add validations that cause an error to our existing parsers unless absolutely necessary for new functionality. In OpenAPI that's less likely to be possible as it is a more rigid YAML/JSON structure and new features would be added in new properties of an object. Perhaps an example of this in OpenAPI 3 (at least for us), is that we don't fully support the complete spec in regards to some of the optional properties. servers in OpenAPI Object is one example of this, if a user uses servers right now they are discarded and the user is shown a warning regarding the unsupported feature. In the future, when we add support for servers, if we introduced an error, it could mean that a previously supported document would emit an error and thus would no longer be parsable on newer parsers.

I wanted to bring this design point from Apiary for discussion and propose that some of these items above that are specified as type error may not need to be errors. For example:

Path parameters definition (Either at the path-level or the operation) need matching parameter declarations

Parser's can implicitly create a parameter definition in its parse result for the path parameter if missing.

References must point to existing documents or document fragments

If the reference is in an optional property (or optional tree of properties) in an object it can be discarded.

[darrelmiller]

We've found cases in the past where users will discover that their (supposedly valid OpenAPI 2.0 document) could be used in other tooling but could not be used in Apiary due to more strict validation

The solution to this problem is not to be more lenient but to encourage tools to respect the rules of the specification. If tooling wants to implement "fix my description" mechanisms they are encouraged to do so. However I would absolutely not recommend that tools attempt to implicitly correct descriptions nor ignore errors. This approach severely hinders learning and leads to poor interoperability.

[handrews]

I'm going to move this to the OASComply project, as it is addressing this sort of thing.