How to validate against Swagger API spec

[lazycodeninja]

I've written up an API specification into a Swagger YAML file. I'm trying to work out how I would validate a response against this, from what I can see the only way I can do this is to code this in a separate Java class and use the Java interop to define a function that calls the method. This looks a little bit messy, is there any other way that I've missed?

[ptrthomas]

If it makes sense Karate can support YAML API-specs in the future. Do you have some example of what exactly you are trying to validate in this case ? But yes, you can write a helper Java utility and get going.

My (opinionated) take on this is that the only benefit you get from this approach is to validate that the payload conforms to some schema. Which won't be an issue for your service 95% of the time because you would be using some POJO / contract-driven framework anyway.

I would just cut and paste the JSON (or XML) from a sample payload you may have already and start writing tests around that.

[rgupta7888]

Hi Peter,

I have a question on using swagger integration with Karate. Let's for example, I have 50 REST API drafted using Karate framework.Now, if some requests of any random automated API's got changed then running Karate will give me definite errors so, I have to modify the API's requests.

We are using swagger in my project as REST API docs and it has updated request parameter so is there any way to identify the services which need to be updated through Swagger before running the test suite to avoid using manual changes in the changed API's using Karate.

[ptrthomas]

This is what you said:

running Karate will give me definite errors

I would say this differently !

running Karate would successfully catch all the changes that could break clients of my API

Any further questions :)

BTW Swagger is actually problematic in practice because it depends on developers keeping the annotations / documentation accurate and up to date.

[haroon-sheikh]

@lazycodeninja @rgupta7888 how did you guys workaround this?

[lazycodeninja]

@haroon-sheikh I didn't end up using Karate. Instead, we wrote tests in Java using RestAssured for given/when/then Gherkin syntax, and a final step in each test to validate the requests/responses against the Swagger file using the Atlassian Swagger Validator. If you're testing a Spring Boot service it ends up being pretty concise as a SpringBootTest (using the annotation).

Sorry Peter I never responded to your message. You care correct, we intend to verify that the producer and consumer(s) of an API send and receive messages defined in the spec. And if the spec file changes, it can be automatically validated against both the producer and consumer to determine whether a breaking change was added to the

Which won't be an issue for your service 95% of the time because you would be using some POJO / contract-driven framework anyway.

I don't quite understand what you mean by this. The spec is effectively the contract, so why wouldn't you want to validate against it?

[ptrthomas]

@lazycodeninja this thread has someone who is building a framework on top of karate to include swagger based validations: https://github.com/intuit/karate/issues/226

I don't quite understand what you mean by this. The spec is effectively the contract, so why wouldn't you want to validate against it?

IMO swagger limits what you can validate to: the payload format and schema and the HTTP method. Many teams that do contract-first would have generated the POJO-s used via some tool, which means you never run the risk of the server messing up on the JSON format. Which brings me to believe that the value-add via using Swagger as a base for your automated tests is low - maybe it does give you some lift to ensure that CRUD is working. I personally would be more concerned about business rules / conditional logic / boundary conditions.

Anyway, YMMV :P

[alvaroacev]

Thank you @lazycodeninja for the reference! I ended up using swagger-request-validator. It demonstrates the usage of the validator with a variety of frameworks, from spring web, mockito, to restassure, useful for those looking to perform test locally (when you know the response) to those doing more kind of elaborated integration tests

[ptrthomas]

for the benefit of those who may come across this thread in future - I'd like to point out the limitations of swagger, best explained here: https://stackoverflow.com/a/34826244/143475

the top 3 which I have seen painfully play out in practice:

• a lot depends on how disciplined the devs are in keeping the swagger annotations up to date
• your java code gets horribly mangled by all those annotations and JSON-s that you need to add in
• representing flows more complicated than retrieving and using an auth token is hard / not practical

and these are the reasons why the spring team created Spring REST Docs: https://spring.io/projects/spring-restdocs

another point worth mentioning is Karate's approach to representing JSON schemas: https://twitter.com/karatedsl/status/878984854012022784 - and many teams are very happy with this super-simple, human-friendly approach to validate payloads. actually most of the time teams don't need to use schemas, they just cut and paste a JSON specimen, tweak a few fields to use #uuid and #string if they are dynamic - and that's far more useful than a swagger spec IMHO

assuming you do manage to keep swagger schemas up-to-date - as I mentioned above in this thread - the value you get from doing this is minimal - especially if you are doing code-gen from something like an Open API spec. personally I would rather put effort into testing flows and business logic - which is what I consider real integration testing.

anyway. we are planning to incorporate capabilities of Spring REST-docs in a future version of Karate: https://github.com/intuit/karate/issues/951#issuecomment-548272879 - and we expect that combined with the API mocking capabilities - we will have a pretty compelling solution for documenting API-s. because if you have this awesome combination of a) tests b) embedded wiki docs c) mocks - these are sufficient to describe any API in full - including headers and Authorization requirements

just my 2 c :)