What about Swagger Specification aka OpenAPI?

[elgalu]

Maybe we are kind of re-inventing the wheel here? see OpenAPI

[mefellows]

Hi @elgalu, this is a great discussion point. For starters, Pact pre-dates OpenAPI (not Swagger), so I like to think we're not reinventing the wheel! But as I understand it there are some key differences and if anything, they are complimentary technologies and could be used together.

The differences can be summarised below:

Swagger / OpenAPI specification aims to standardise the description and structure of an API. It can tell you what APIs are available and what fields/structure it expects and can generate documentation/UI to interact with one. What it is not, is a testing framework.

Pact on the other hand, is essentially a unit testing framework using specification by example. It just so happens that to be able to run those tests on the API consumer and provider side, it needs to generate an intermediate format to be able to communicate that structure - this is the specification. Now we need a lot more information that just the structure (matching rules, provider states and so on) that OpenAPI documents in its spec.

In fact, the authors of the OpenAPI specification predicted such use cases by announcing:

Additional utilities can also take advantage of the resulting files, such as testing tools.

Potentially, for example, we could use vendor extensions to document this extra metadata that is captured in our spec. This is one way the two projects could come together.

So for me, I would like to explore further how we could use existing Swagger/OpenAPI definitions to streamline Pact usage - for example to generate stub tests or as part of our specification. Suggestions most welcome :)

Does this help?

[elgalu]

Awesome answer @mefellows, my question should have been more like How Swagger and Pacts could complement each other

To add some background, at Zalando we are pushing Swagger quite a lot and we have some related open source projects: connexion , play-swagger , swagger-mock , codegen-tooling

explore further how we could use existing Swagger/OpenAPI definitions to streamline Pact usage

That sounds great! maybe we can use this thread to communicate our findings.

[mefellows]

Glad to hear we're on the same page. I'm OK with the conversation staying here, but would like to here thoughts of @uglyog and others. It might be best to move to a Google groups discussion.

[uglyog]

I have done quite a bit with both Swagger (or OpenAPI) and RAML. Although they seem to serve the same purpose, their use cases are slightly different. Swaggers main use is as a design tool, allowing the APIs to be specified and then the server and client parts to be generated off the specification. At my current client, all APIs have to have their solution design signed off before they can be built, and swagger serves this purpose well.

Pact supports a more evolutionary development process, and I have used it in many Agile projects where the APIs have evolved from the requirements of the user stories. It was always designed from the start to have the contracts generated from actual running code, so the contracts are always current. Swagger and RAML work the other way round, the code is generated off the specification and there is no automatic way to keep them in sync.

The other problem is that Pact is "Specification by Example". It is a record of example request response pairs that determine the specification. Swagger, OpenAPI and RAML are all based on paths, and normally have a single request specification and then multiple response specifications for each different kind of response. I have had discussions on how to generate the specification from the pact files, but it turned out to be difficult because with pact you can have multiple examples for the same API path, so which one do you use for the specification?

[jfnavin]

To contribute to the conversation - our team is experimenting with validating the Consumer-generated Pacts against a Provider-published Swagger/RAML specification within the Consumer tests.

We've found it can help catch a class of invalid expectations very early (bad paths, incorrect parameter formats, request/response schema validation etc) and shorten the feedback loop (you don't have to wait for the Provider tests to run to realise you've made a mistake setting up your expectations).

It does come at the expense of requiring Consumers to set up expectations that include all required fields in a request/response schema rather than just the ones they care about (breaking Postel's law), but at the moment that hasn't been a big problem for us.

We're also looking at the possibility of generating REST clients from the Swagger/RAML spec that include a 'mock' mode powered by Pact files (havent got very far with that spike though...)

[uglyog]

@jfnavin I'll be really interested to see what you can come up with.

[cah-andrew-fitzgerald]

Had another sort of similar-ish project to add to the conversation. Spring REST Docs feels like it's somewhere in between swagger and pact consumer tests.

The short version is that you write tests describing your endpoints, and as a side-effect of running those tests new documentation is created.

[jfnavin]

@uglyog - Ive (finally) open sourced our swagger-request-validator library that includes a Pact module for validating Pact interactions against a Swagger/OpenAPI spec.

https://bitbucket.org/atlassian/swagger-request-validator

There's some examples in there on usage etc. The library is still very much under development - if you find any issues raise a ticket on the project and I'll try to look at it.

[uglyog]

Yay! :metal: I'll definetly give it a look.

[mefellows]

@jfnavin this is interesting! Also, are Atlassian using Pact? If so, we'd love to hear about how/what/etc.!

Can discuss here or better yet: https://gitter.im/realestate-com-au/pact

[BenSayers]

We'll be sharing some details during Atlassian Summit 2016 so come join us if you can! https://www.atlassian.com/summit/sessions?tab=build-%26-deploy&sessionid=54094

[mefellows]

Wow, that's awesome! Maybe we'll see if DiUS can fly us to California to see it in person ;)

On a side note, would you be interested in talking (privately) to us about your travels with Pact at Atlassian? We'd love to hear about what you're doing and what we can do to improve the product.

On Fri, Sep 2, 2016 at 3:50 PM, Ben notifications@github.com wrote:

We'll be sharing some details during Atlassian Summit 2016 so come join us if you can! https://www.atlassian.com/summit/sessions?tab=build-%26- deploy&sessionid=54094

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pact-foundation/pact-specification/issues/28#issuecomment-244288994, or mute the thread https://github.com/notifications/unsubscribe-auth/AADSjB96ay9qBbb2bmdQhDhaKuOEiVQcks5ql7kWgaJpZM4IiRod .

Matt Fellows

• http://www.onegeek.com.au/

[fitzoh]

@BenSayers is there a recording or some slides out there?

[BenSayers]

@Fitzoh here is the recording - https://www.youtube.com/watch?v=-6x6XBDf9sQ

[zouroto]

Hi there, I'm using pact swagger validator and it works well. But i try to put some response exemple in my swagger file, and it seem's that the validator take in account only the structure of the openApi file and not the data i put inside the response example part. Could you tell me if it's possible to validate Pact with OpenAPI with examples data ?

"responses": {
          "200": {
            "description": "successful operation",
            "examples": {
              "application/json": {"firstName":"random", "lastName":"mandatoryValue", "id": 1, "status":"CLOSED"}
            },

[BenSayers]

@zouroto what would you expect the validator to do when comparing the pact to the example in the openapi file?

The only thing I can think of is it could check to see if the pact and the example exactly match each other. However I’m not sure how useful that would be. It’s common to test the same endpoint multiple times, stimulating different responses. For example a 200 with no items in an array and a 200 with multiple items in an array. Looking for strict matches would prevent people from writing multiple cases for the same endpoint.

Perhaps I’m missing something?

It might be worth mentioning that while the validator does not look at the examples, it does look at the schema if you define one. Is there anything preventing you from describing your response format using the schema section?

[zouroto]

Hi, thank you for your response.

I've no concret example for a such use, i'm just trying to understand what is good to do with the swagger validator or not. For exemple with the pact DSL we can use "stringValue" if we expect a specific value to be send by the provider. In a pact provider test (without swagger validator) we start the provider as a black/grey box then use provider state to insert some data, and as a result of a call we could expect a specific value to be present in the response.

Maybe this kind of scenario does not exists with the use of swagger validator on provider side.

A final question : is there any support for Junit 5 ?

[BenSayers]

In my organisation we have thousands of micro services and hundreds of teams involved in operating them. We ran into a number of problems when trying to use the full Pact validation approach with this many teams and services:

• When a consumer wanted to add a new test requiring new provider state they would become blocked waiting on the provider team to add the state they required. This could sometimes take weeks to resolve if the two teams were in a different time zones.
• When a consumer published a new Pact file we wanted to automatically execute the provider validation as part of the build of the consumer. However not every team has the same level of build reliability, so when a consumer who has very strict reliability requirements on their build integrates with a provider who does not, this creates problems. We also never found a way to standardise the builds enough to make this orchestration simple and low cost to initially setup.
• We ran into a similar issue around misaligned build speed expectations between teams that also created problems. Small fast micro services don't want to wait 45 minutes for a slow legacy provider to validate their Pact files.

These problems led us to the swagger validation approach, which addresses all the issues we ran into. The tradeoff is that the swagger validation is not as strict as the full Pact validation, and you need a way of ensuring that the swagger file accurately reflects the behaviour of the server. I'll leave it to you to decide which tradeoffs make sense for your use case.

Try asking your JUnit question as an issue on the specific project/library you are asking about.

[dwang7]

@BenSayers if you are validating the pact files on the consumer side with the provider swagger file. Do you actually do any validation on the provider side using the consumer pact files or are you doing all your validation on the consumer side only? Do you run any other contract tests on the provider besides the swagger validation?

[BenSayers]

@dwang7 yes we use the swagger mock validator on the provider side as well. We take the newly generated swagger file and validate it against all the consumer pact files for that provider.

[dwang7]

@BenSayers Do you do any additional validation on the provider side besides the swagger validation?

[BenSayers]

@dwang7 yes we also use a seperate tool we’ve written called openapi-diff that can detect when breaking changes have been made to a swagger/openapi file: https://bitbucket.org/atlassian/openapi-diff/src/master/

[timonbimon]

@BenSayers the talk you linked here is incredibly good! I'd be really interested in the "state of contract testing @ Atlassian" now two years later (by two years I guess I'm referring to the comment here where you mentioned some problems you had with the full pact validation approach, not your talk). :)

[Nashev]

Why can't we generate summary-like OAS (swagger) specification for provider by all amount of accepted consumers pacts and then use it as a draft for manual correction, and after that generate by this spec an implementation of a server, clients, documentation, verification proxy, some additional tests and so on?

If OAS generation by pacts will be fairly stable (regarding the order in which paths, arguments, and structures are declared), I think it will be very useful.

[mefellows]

There's nothing stopping you building this now @Nashev! As the pact specification and OAS are documented formats, you could write a tool that can collect one or more pacts, combine them in some reasonable way and then generate a spec.

There will be complications, however. It might be very hard to infer polymorphic endpoints and the use of keywords like anyOf, oneOf etc.