search

operator-framework / operator-courier

Build, verify and push operators
Apache License 2.0
41 stars 53 forks source link

Add basic integration tests with behave #73

Closed jeremy-wl closed 5 years ago

jeremy-wl commented 5 years ago

The integration tests currently only test the verify command. For the push command, we need to set up quay access token as environment variable. Due to some restrictions from Travis CI,

Encrypted environment variables are not available to pull requests from forks due to the security risk of exposing such information to unknown code.

Similarly, we do not provide these values to untrusted builds, triggered by pull requests from another repository.

We decided to include it later and run those tests directly on master, and trigger Travis CI builds periodically with cron jobs. Details are yet to be decided.

jeremy-wl commented 5 years ago

/cc @awgreene @kevinrizza @SamiSousa

csomh commented 5 years ago

Ok, I'll be the devil's advocate, again :)

I strongly discourage using behave for the purpose of integration testing, as it's not really suitable to test these kind of applications on the long run.

I had a really bad experience setting up an entire integration test suite for OSBS, which was based on behave, and which was ultimately migrated to pytest in order to improve maintainability and ease further development.

Behave introduces a layer of abstraction which is hard to work around when testing APIs or CLIs. Behave is suitable for testing applications or services, where the functionality can be described in relatively simple human terms. Operator Artifacts are beyond that, I think, and as such the language of the scenarios will become convoluted and hard to follow.

jeremy-wl commented 5 years ago

Hi @csomh, thank you very much for your advice. Could you provide some concrete examples or scenarios why behave might be poorly suited for operator-courier? I current don't have issues with this framework as it does what I want it to do, and switching to something else would take extra effort.

Also, I have set up a group chat on slack with @kevinrizza, feel free to reach out to us there to discuss in depth.

jeremy-wl commented 5 years ago

Closing it because we decide to use pytest instead of behave for integration tests.