Create end-to-end tests for openAPI

[drasko]

A set of e2e tests should be created to check the sanity of the system before merging new PRs.

Test should:

• Act as user - use SDK to register and login user
• Use SDK to provision things and channels
• Act as a Mainflux thing (device or app) - use SDK to send messages that will be written via Writters to various DBs (C*, InfluxDB, MongoDB)
• Use SDK to read messages from various Readers
• Compare and validate read messages

This way whole Mainflux system will be treated as a blck box, and tests will proven API sanity and that main functionalities are preserved.

[alwindoss]

e2e tests are usually run on a deployed software. Are you hinting towards a deployment on docker to run the e2e tests and then the developer can raise a PR?

[mandric]

Is this something a newbie could work on? I have experience with e2e testing tools in the NodeJS world and little Golang experience but am motivated to learn.

[drasko]

@mandric yes, definitely you can give it a try. I think it should start by selecting a correct technology. @chombium and @anovakovic01 mentioned Pact before, and there is a Go version: https://github.com/pact-foundation/pact-go

I am not sure at this point what thecnology would be best to do this, some research on technology choices is needed.

[mandric]

Pact looks interesting, I am not a Ruby person either, I have Python experience though. But if you give me the set of tools I can play around with it and see how far I get.

[mandric]

There appears to be python version, not sure if that's interesting. https://github.com/pact-foundation/pact-python

[mandric]

Oh sorry, it would obviously be written in Go! That could be a fun way to learn!

[mandric]

Well I looked a little at pact-go and the APIs seemed a little complicated, so I just wrote something up that was quickest for me using bash and bats. I thought I would post here to get some initial feedback. I figured it would be useful even if not as a long term solution but maybe as a way to quickly get some coverage and maybe help think about the problem. I would be happy to go back to pact-go but it would take more time to get something up and running.

https://github.com/mandric/mainflux-e2e-bats

The next steps if you wanted to continue on the bats path would probably be 1) obviously adding more coverage 2) hooking up to CI.

[mandric]

This might be a better option that has more automation using swagger, postman and newman: e.g. https://github.com/gothinkster/realworld/tree/master/api

I haven't researched yet but I imagine GoLang has equivalent toolset.

[chombium]

@mandric this looks really interesting. I guess we'll have to add some glue code which will go through all of our swagger files and generate the whole api spec + some test data, before testing the api itself

[drasko]

Yes, this is interesting. I researched already go-swagger before.

In any case, fining an equivalent to this would be interesting: https://github.com/postmanlabs/newman

[nmarcetic]

Check out Kubernetes methodology overview Also Ginkgo Looks great.

[nmarcetic]

@mainflux/maintainers I took this issue to do research.

[drasko]

A Peorperty-Based Testing technique is worth investigating: https://www.youtube.com/watch?v=hXnS_Xjwk2Y

Haskell has a famous tool QuickCheck, maybe some libs/frameworks for Go exist:

• https://blog.gopheracademy.com/advent-2017/property-based-testing/
• https://github.com/flyingmutant/rapid
• https://github.com/leanovate/gopter

[darkodraskovic]

I did a small research using links posted here. Here are my conclusions concerning the e2e testing possibilities.

Property based testing

A property based testing is a testing paradigm that is, as far as I understand, used mostly for unit tests. It is about trying to avoid thinking about all the possible cases - which is an empirical and ad hoc approach - and to implement a more general or systematic way of testing. This is the basis:

Instead of asking “what is the expected result of this program given these example inputs?”, we ask the question: “what is the property of the result and program that doesn’t change given inputs?”

So for example, we can test Add function with Add(x, y) and Add(y, x) expecting the result would not change, because our function is commutative. That said, the property based testing is primarly for unit tests: we target a singular function.

Behavior-Driven Development

Behavior-Driven Development is an extension of test-driven development and can be used indiscriminately for any kind of tests. That said, it is more a methodology or a paradigm than a concrete set of tools. It's like a certain style of programming that can be used with many different languages. Actually, https://github.com/pact-foundation/pact-go encourages this kind of development, as far as I understood.

https://github.com/onsi/ginkgo is based around the Behavior-Driven Development approach. However, it is meant to be used for unit tests, rather than for e2e tests.

Check frameworks

Frameworks like https://github.com/leanovate/gopter, https://www.scalacheck.org/ and http://hackage.haskell.org/package/QuickCheck are used for "for random testing of program properties" and they are basically test data generators, so you don't need to enter input test data manually. That said, they are per se not meant for e2e testing and suit best for unit testing. However, they can be used to generate random data for requests and responses for e2e tests.

Pact Go

"Pact is a contract testing framework for HTTP APIs and non-HTTP asynchronous messaging systems." (https://github.com/pact-foundation/pact-go) Pact is, AFAIK, the only widely adopted tool written specifically in golang that can be used to deploy e2e tests.

It treats services as providers and consumers and makes a mock provider for consumer and mock consumer for provider, so you don't need to launch a composition of services, i.e. you don't need to deploy in order to test. In order to achieve this, it uses the so-called contracts, where we store the metadata about valid consumer requests and valid provider responses in the separate files, namely, contracts. These contracts have to be updated from time to time in order to synchronize them with the state of the service development.

This approach stimulates BDT as described here https://en.wikipedia.org/wiki/Behavior-driven_development. For example, a consumer team might make assumptions about provider service and make a contract for testing consumer service independently of provider service. However, it needs to communicate with provider service in order to check for the pertinence of the contract. This opens a negotiations process, etc.

Postman alternative to Pact Go

The only alternative that I found is using Postman automation to write e2e tests. In itself, it is not bad, however, we add another tool to our toolchain as well as a closed source application which is, according to our standards, not acceptable.

Conclusion

We should use https://github.com/pact-foundation/pact-go for e2e tests and eventually add generators ("check" frameworks) for the test random data. However, we should start with the plain Pact. As an added bonus, we do not need to use any other languages or libraries to use Pact, since the go version of the Pact is ported to go and does not have Java or any other dependencies (someone correct me if I am wrong).

[drasko]

Thanks @darkodraskovic this looks good.

Take care that Mainflux API is not only HTTP, but also async (event-driven) API (for messaging - for example MQTT). Luckily, Pact seem to support that: https://docs.pact.io/implementation_guides/go/readme/#asynchronous-api-testing

Some interesting articles:

• https://www.serverless.com/blog/dave-copeland-contract-based-testing-event-driven-architectures-emit-2017
• https://medium.com/@dius_au/contract-testing-serverless-and-asynchronous-applications-6e614e301fc1
• https://medium.com/@dius_au/contract-testing-serverless-and-asynchronous-applications-part-2-76b5576c789e

It will be useful even for gRPC contract testing - as we have gPRC APIs as well between services (for example Auth), and we plan to expose gRPC API in the future for integration:

• https://medium.com/@ivangsa/consumer-driven-contract-testing-for-grpc-pact-io-d60155d21c4c

[drasko]

Additional docs to read:

• https://medium.com/@m_arlandy/contract-testing-for-microservices-using-swagger-prism-and-dredd-efdd463b9433
• https://dev.to/waleedashraf/asyncapi-for-documentation-and-validation-of-event-driven-architecture-2ap8

They look interesting - especially the second one, as they can use existing Swagger, OpenAPI and AsyncAPI files (yamls) to generate tests and mocks.

I would really like that we stay tied to a contract defined in Swagger / AsyncAPI spec and reuse this existing spec to generate tests. I do not know how Pact can do this - I guess it can.

[drasko]

Related to this: https://docs.pact.io/faq/convinceme/#but-i-use-swaggeropenapi

[drasko]

https://github.com/swaggest/go-asyncapi

[drasko]

https://openapi.tools/#testing

Interesting are tools for generating automatic tests from Swagger, like this one: https://github.com/schemathesis/schemathesis

I am afraid that Pact demands it's own spec format: https://github.com/pact-foundation/pact-specification/issues/28, not compatible to our existing Swagger / OpenAPI / AsyncAPI formats...

Thus - if we go with Pact - I am afraind that we will not be capable to leverage on our existing Swagger specs, but will be forced to create and maintain duplicate of specs, in Pact format.

@dusanb94 @nmarcetic @manuio @mteodor @blokovi what is your opinion on this? I do not like it, TBH. I think it will be hard to maintain double specs, but maybe there is some gain (Pact over Swagger generated tests). Is this gain worth the pain of maintaining double specs?

[drasko]

Very interesting: https://arxiv.org/pdf/1912.09686.pdf - this is what I had in mind with Property-Based Testing:

1. Use Swagger / Async API to generate tests
2. Randomize paramters a la QuickCheck
3. Profit

[drasko]

Related to https://github.com/mainflux/mainflux/issues/1149

[drasko]

• https://medium.com/@you54f/protecting-your-api-development-workflows-with-swagger-openapi-pact-io-5ba8dca8b9cf
• https://pactflow.io/blog/the-curious-case-for-the-provider-driven-contract/
• https://github.com/pact-foundation/pact-specification/issues/76
• https://docs.pact.io/faq/#can-i-generate-my-pact-file-from-something-like-swagger
• https://groups.google.com/g/pact-support/c/pNaz1mN9fZQ?pli=1

[drasko]

TBH, having all this in mind, I am leaning mostly to something like this:

• https://www.youtube.com/watch?v=1lo7idI7uq8
• https://github.com/schemathesis/schemathesis

As I mentioned before, I really like Property-Based Testing approach - as referenced in https://github.com/mainflux/mainflux/issues/643#issuecomment-687667599

Nice articles about Schematesis' PBT approach:

• https://code.kiwi.com/schemathesis-property-based-testing-for-api-schemas-52811fd2b0a4
• https://dzone.com/articles/schemathesis-property-based-testing-for-api-schema

Probably this nice Haskell tool is also worth exploring: https://github.com/rodrigosetti/swagger-test

Having in mind Mainflux lightweight mindset, I would not be surprised that this Haskell tool would be the best fit :). Just that it supports OpenAPI-only, and we'll need AsyncAPI as well (for MQTT purposes) - read more about it here.

[Stranger6667]

Hello, all! The Schemathesis author here. Happy to see that you are considering Schemathesis :) Let me know if I can provide any support from my side, e.g., missing features or something not working as expected :)

[drasko]

@Stranger6667 thanks for your work on Schemathesis, I think it is ideated in the right directions, alongside the idea of PBT and QuickREST article.

One thing I see now missing for our purpose would be AsyncAPI, bacause in IoT MQTT is a predominant protocol, and in general lot of PUB/SUB paradigm is used.

Is there any plans to add this support from your side, and how hard would be to add it?

[Stranger6667]

Thank you!

Is there any plans to add this support from your side, and how hard would be to add it?

AsyncAPI is new to me. As far as I see, this spec is very close to Open API, and it should be relatively easy to adjust the existing Open API code to generate messages or other structures since they follow the JSON Schema spec (as far as I see).

However, to support a complete testing flow we need wider transport support. Schemathesis is centered around HTTP, and building other transports might require some more investigation and time. I will take a deeper look there, but from my side, AsyncAPI is definitely something I'd like to support in Schemathesis :)

At the moment I am working on making Schemathesis more spec-agnostic tool and available as a service - not sure about the bandwidth I can allocate now, but I think that supporting AsyncAPI would be a logical next step for Schemathesis after I am done with refactoring :) I expect that I'll have time to bring some proof of concept prototype during October

[drasko]

Adding John Hughes talk (on QuickCheck) for the reference: https://www.youtube.com/watch?v=zi0rHwfiX1Q

And another one: https://www.youtube.com/watch?v=hXnS_Xjwk2Y