feature: streaming checks

[nbgraham]

I recognize that this is a big change and may not be accepted, but wanted to get the idea out there. I'm pretty sure it's not a breaking change though.

I started looking at using this library with gRPC health checks but couldn't figure out how to utilize the stream option where the server pushes status updates.

So I made this PR as an example of "streaming" check: a check that listens for updates on a go channel (dependency pushes health check updates instead of this app pulling them).

This "streaming" check is a more generic form of the periodic check. So I implemented the periodic check as a streaming check internally. A streaming check could be built upon for extremely custom async checks like

• while healthy, check every x seconds
• on first fail, check more frequently on exponential backoff

Other tweaks

• made separate, read-write mutexs. This might be more performance but mostly for clarity.
• allow setting the root context of the async checks

Comments

• a periodic check with a period of 0 was literally indistinguishable in code from a sync check. I found this really confusing when working with the unit tests. With this PR, that is no longer true.

[alexliesenfeld]

Thanks for providing this PR!

I think your idea is interesting and code looks fine at a first glance as well. I am not sure at the moment if streaming or gRPC should be supported by this library. The initial codebase was written to support standard HTTP and I think adding streaming or gRPC would require a substantial redesign for a robust implementation.

I'd like to park this PR for now and revisit it if gRPC becomes a thing.