Actual integration testing on CI

[peterbourgon]

Go kit includes some packages which are designed to interact with popular open source software. They include

• sd/etcd — with CoreOS etcd DONE
• sd/consul — with HashiCorp Consul DONE
• sd/zk — with Apache ZooKeeper DONE
• tracing/opentracing — with AppDash, Lightstep, and Zipkin

Many of those packages include proper integration tests that connect to the relevant software. (Some do not.) In this ticket we should

• Provide simple, reliable integration tests for the packages above that don't have them; and
• Configure one or more of our CI servers (Travis, Circle, and/or Drone) to run those tests.

Writing the tests should involve only basic connectivity and expected functionality. No need to get clever checking for obscure edge conditions, the big win here is making sure we don't regress.

Running the tests should involve absolutely minimal changes to this repo: no complex setup or teardown scripts embedded into circle.yml, please! Leverage Docker images where appropriate, and go with standalone or single-node test-cluster setups wherever possible.

[fvosberg]

Hi Peter,

I would love to help you with this issue, but I don't really get what I can do. As I've seen there are basic tests in the packages. Is there a specific case, which you are missing? Or should I configure running tests on a specific server?

Thanks in advance

[peterbourgon]

Hi @fvosberg, thanks for reaching out! You're right, the state of the actual project has accelerated away from this issue, at the moment we have good integration tests implemented for all of sd/{etcd,consul,zk} and running on Circle. I'll update the original issue to reflect that.

The only thing left is the tracing stuff, which is tricky! It's slightly less necessary, because OpenTracing itself encapsulates the vast majority of the hard stuff, and it's already well-tested internally. It's also a bit less obvious, because a useful integration test from Go kit's perspective would involve quite a lot of work—though I think it could be quite illuminating!

Off the cuff, here's what I think would need to happen for the integration test, in rough strokes:

• Pick an OpenTracing-compatible system to test against
• Create 2 simple microservices, A and B, fully instrumented for that tracing system
• Make sure that A depends on B, i.e. A calls out to B in one of its methods
• In the integration test, boot up all 3 systems and ensure connectivity
• Make a request to A
• Wait some time, and then query the OpenTracing system for the trace
• Make sure everything is there, and the observed timings are within some tolerance of expected

The nice thing about this work is that it's nicely self-encapsulated. It's its own little universe of things, probably running in one or more Docker containers, not a lot of touchpoints with the outside world. And I bet it would serve as an excellent example to others!

Is this something you would find interesting, and capable of attempting?

[peterbourgon]

I'm gonna call this resolved. OpenTracing integration tests would be super nice to have, but I think quite difficult to make worth the while.