Certification program for OAS consuming implementations

[tedepstein]

Some organizations considering adoption of OpenAPI have asked what assurances we can offer with regard to interoperability.

• They want to see evidence that the OpenAPI specification itself is mature enough, and well-enough specified, that tools and libraries can be tested for conformance.
• They want to see a commitment to interop and specification conformance on the part of OAI as an organization.
• When considering adoption of specific tools, they want some objective evidence that those tools conform to the OpenAPI specification.

This issue builds on some early discussions in OAI/OpenAPI-Specification#1520, which was originally scoped to defining a format for the rules and test cases that comprise a test suite. That is one aspect of the program.

This issue is about the first phase of the overall certification program. The TSC has agreed that this first phase should focus on OpenAPI consumers, i.e. implementations that take an OpenAPI document as an input.

[tedepstein]

Here's a strawman proposal for a milestone plan:

M0. Suggested Next steps:

• Example YAML file and schema for validation.
• Provision some resources in the current OAI project, or create a new one.
  • YAML data document with initial subset of rules and test cases
  • JSON Schema describing the format of the YAML doc.
  • Readme markdown document describing:
    • Purpose of the conformance test Suite
    • Scope (consumers only for now)
    • Current status (experimental, draft, comments welcome)
    • Available resources (YAML data document with rules and test cases)
    • Plans

M1. Prototype Test Automation Tool

• Complete a subset of rules and test cases
• Prototype test automation tool to inject each test input & inspect response.
• Prototype test adapter for one tool, same platform as test driver (e.g. Java)
• Review with TSC

M2. Test Automation Tool RFC

• Evolve subset of rules & test cases
• Develop 1 or 2 additional test adapters for other tools, other platforms.
• Add to the readme: overview of the certification process:
  • How to start, by self-verifying conformance.
  • How to raise questions & issues.
  • How to request review & certification from the TSC (or certifying authority, whoever that is.)
• Package and polish the the readme file and related artifacts (github project under OAI recommended), with a CTA to providers of OpenAPI consuming tools & libraries:
  • Please look at the conformance test suite and the automated testing tool.
  • Please try building your own adapter & running the test suite.
  • Please give feedback!
• Limited publicity and/or targeted outreach to OAS consuming tool providers.

M3. Release Candidate

• Complete the rules & test cases.
• Incorporate feedback into the process, readme, test automation tool & adapters.
• Make sure readme and process management tools (if any) are complete.
• Broader publicity/outreach to providers.
• Run provisional certification process with participating providers.
• Add certification status to implementations page, with provisional cert icon for those implementations.

M4. Certification Program Go-Live

• Update readme with live status
• Remove provisional qualifier from certified implementations
• Publish updated roadmap for future phases of the certification program.
• Public announcement/publicity.

[handrews]

@swaldron58, @dret should this issue be transferred to the Outreach repo?

[dret]

On 2024-01-27 04:04, Henry Andrews wrote:

@swaldron58 https://github.com/swaldron58, @dret https://github.com/dret should this issue be transferred to the Outreach repo?

that sounds like a good idea. thanks!

[swaldron58]

Sure, go ahead. I don't have the authority.