search

openservicebrokerapi / servicebroker

Open Service Broker API Specification
https://openservicebrokerapi.org/
Apache License 2.0
1.19k stars 434 forks source link

Conformance test suites and the OSBAPI Working Group #653

Closed mattmcneeney closed 5 years ago

mattmcneeney commented 5 years ago

One of the goals of the OSBAPI group has been:

Create a conformance test suite to verify both consumer and producer behaviours

Whilst the group adopted the osb-checker from Microsoft in August 2018, the group has found it hard to maintain this test suite (which is focussed on verifying service broker compliance with the spec). As a result, the test suite does not support the latest version of the specification, and we have seen other test suites emerge in the community.

When adding new features to the spec, there is already a considerable amount of work that has to be done by the community, including validating implementations, updating the spec and updating multiple swagger/openapi documents. Updating a test suite which, for many of us may be written in a programming language we are unfamiliar with, as part of every change to the specification seems to be unfeasible.

@fmui and I have been talking about this at length, and we'd like to propose a couple of changes:

  1. We remove the goal from the https://www.openservicebrokerapi.org website about creating a conformance test suite.
  2. We enhance the project's getting started page with a list of test suites which service broker authors could use.

The test suites that support the latest version of the spec and are being maintained today include:

  1. osb-checker-kotlin by the team at Evoila including @chbrinker
  2. osb-checker, which was originally donated by @arschles at Microsoft but has subsequently had a lot of work put into a redesign by @leonwanghui

There may be more test suites out there, and like the various libraries we link to, we could simply link to these test suites for folks to choose from. This list can easily be extended through pull requests to the getting started page in this repository.

Before proceeding with this, we'd like to get your feedback and will be discussing this on the next OSBAPI call on the 9th April.

leonwanghui commented 5 years ago

Before expressing my opinion, I think we all agree that the conformance test is very important for standard specification. If so, the question would be who should be in charge with the maintenance of conformance test, To me I prefer to maintain it in the community.

Firstly there are some reasons why I strongly suggest keeping that goal:

For the issues pulled by @mattmcneeney, I think we can leverage some open-source tools to reduce the development complexity as much as possible. For example, we can use some open-source behavioral test tools and code-generator to reduce code work, and make users run it everywhere by building binary file or docker image.

Lastly, as member of this community, Huawei considers the conformance test as important part of OSB API standard. Since several Huawei projects (both producers and consumers side) has adopted this specification and we also have embedded the osb-checker into our OpenLab pipleline, we are willing to contribute in the long term and maintain the code in the community.

waterlink commented 5 years ago

I agree with @leonwanghui

Does it make sense to pull in the test suite into the Spec repository and require contributors to the spec also to alter/add conformance tests in the same PR as the change to the markdown file?

(if that’s possible, it seems to be an ideal solution).

mattmcneeney commented 5 years ago

Thanks @leonwanghui that is very useful feedback that we should talk about.

@waterlink - the problem I outlined a little bit above is that it already takes a lot of work to get a PR into the spec, and asking contributors to also change a test suite in a language they may be unfamiliar with is a high bar.

leonwanghui commented 5 years ago

@waterlink In the ideal world it's a wonderful idea. But in the real world we have to make some sacrifice for that goal. @mattmcneeney So my suggestion is that can we keep spec and conformance test separately, but require the contributor (or others in the community) to pull an issue in the test code repo when new features are added in the spec? In that way, we can let experts do their own job.

waterlink commented 5 years ago

The language bit makes sense. Have we considered any language-agnostic API testing tool (eg, karate-dsl)?

(that suffers from the problem that contributors will have to learn the tool, of course. It may be easy enough though)

leonwanghui commented 5 years ago

Although the language-agnostic API solves the language problem, but it will bring into one new language which nobody is familiar with.

Actually if we think it deeper, the conformance test is supposed to designed for developers to easily test their brokers, and if I get it right karate-dsl can't relief developers the code burden, so I'm afraid that users would not be interested in something they can't even directly benefit from. So the idea of language-specific implementation would actually work. The first goal is that we should let people accept the idea of conformance test and use that project to validate, and the second goal is to decide if we should design a language-agnostic API for conformance test based on the feedback of those end-users.

waterlink commented 5 years ago

Oh, I see what you are saying. I didn't know that having language-specific support for developers to just drop the test suite as acceptance suite in their codebase was a goal.

If it is so, then I entirely agree with maintaining multiple test suites that will be easy to integrate in the developers' codebases.

Should this then be a selling point on the website pages? Also, what team(s) will be responsible for maintaining it?

It's a fantastic idea 💡 and now there are so much questions to think about!

mattmcneeney commented 5 years ago

I'm very supportive of the goal to help service authors easily test their service brokers for conformance. That seems important for everyone in the community if we are to continue building a diverse marketplace of services for any OSBAPI-compliant platform.

We have learnt that having the OSBAPI PMC be responsible for maintaining these test suites and adding support for every change that goes into the spec is not feasible (oftentimes these changes are optional features which increases the complexity of adding test coverage). This is part of the reason the openservicebroker/osb-checker repository has been left behind and does not support the latest version of the specifcation - not everyone in the community can help develop that test suite (and the same thing would have happened if the test suite was written in another language).

Some possible options:

  1. Change the goal from Create a conformance test suite to verify both consumer and producer behaviors to something like Help service authors find test suites which they can use to verify OSBAPI compliance. We could then link to a number of test suites from this repo.
  2. Keep our current goal and have a dedicated set of contributors who can commit to maintaining a test suite which is hosted by the OSBAPI PMC in this repo. This keeps the bar low for contributing to OSBAPI, but requires commitment from another group to keep the test suite up to date.
leonwanghui commented 5 years ago

I like the second idea, and also be willing to maintain it. But some tiny questions:

mattmcneeney commented 5 years ago

Is the conformance test hosted in the same code repo with specification?

I'm not sure it matters, as long as it's easy to find. But having it in the same Github org is probably sensible (where osb-checker lives today).

Do we need to add more committers/core reviewers to in charge of this conformance test?

Yes, definitely. We'd need folks who have some experience with the particular language and frameworks the test suite is written in who, like the PMC for the specification, would be responsible for reviewing PRs and responding to issues. Ideally this group would consist of at least a few people from a few different companies, so that there are enough contributors to keep the test suite healthy and up to date.

leonwanghui commented 5 years ago

I'm not sure it matters, as long as it's easy to find. But having it in the same Github org is probably sensible (where osb-checker lives today).

Great!

Yes, definitely. We'd need folks who have some experience with the particular language and frameworks the test suite is written in who, like the PMC for the specification, would be responsible for reviewing PRs and responding to issues. Ideally this group would consist of at least a few people from a few different companies, so that there are enough contributors to keep the test suite healthy and up to date.

Double great!!

mattmcneeney commented 5 years ago

@leonwanghui - could you send me a list of folks who would be responsible for maintaining osb-checker, responding to issues, keeping it up to date with the spec, etc?