search

goss-org / goss

Quick and Easy server testing/validation
https://goss.rocks
Apache License 2.0
5.6k stars 472 forks source link

Integration tests fail intermittently #622

Open petemounce opened 4 years ago

petemounce commented 4 years ago

Describe the bug

Our tests have external dependencies

How To Reproduce

Keep retrying.

Expected Behavior

Tests don't fail for external reasons.

Actual Behavior

If an external dependency is unavailable, our tests fail.

Environment:

petemounce commented 4 years ago

httpbin

Used within integration-tests/goss/goss-shared.yaml.

Could replace with a trivial golang http server.

dnstest

Used within integration-tests\goss\goss-shared.yaml.

Could replace with a https://coredns.io/ - I don't know very much about DNS server choices, but this looked at a glance easy to configure.

Orchestration

We'd need to spin up both dependencies, then run the integration tests, then tear down the dependencies. I think that could be achieved via a docker-compose.yml with some judicious port-maps.

I think that could form the basis of reworking the linux-based integration tests, since each test-container could maybe become registered into the compose file - I think that might simplify away a bunch of shell script & makefile.

Approach

If I start this before someone else does (please say hi!), I think I'd

  1. write the httpbin dep
  2. wrap the docker-compose.yml around it
  3. wedge in the docker-compose up --daemon to the integration-tests/test.sh at the top, and tear it down in a trap ... EXIT

That would get us to a working loop, even if it's inefficient to spin up and tear down per test-suite rather than for the whole CI run.

I'd probably then

  1. add the dnstest slice
  2. figure out if I could refactor the Makefile/test.sh to remove the per-suite dependency and turn it into whole CI run.
    • but I'd only do that if it seemed worth the complexity for the speed win
    • ... and really, I'd want to do more in-depth surgery to the integration tests approach, but I haven't got a handle on exactly what yet.
aelsabbahy commented 4 years ago

One thing I would be interested in before actual work is done on this is a brief research on the pros/cons of using docker-compose vs https://github.com/testcontainers/testcontainers-go

The later may be less flexible, but one less build dependency to worry about? shrug

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

aelsabbahy commented 3 years ago

Marking this as approved.

aelsabbahy commented 1 year ago

Status update: