NRPE support

[dselans]

Not sure what all of this entails, leaving a ticket just to keep this in mind.

Would be nice if we were able to piggy back off of existing NRPE configs. Does this mean we implement a NRPE receiver? I don't recall how NRPE works exactly but I recall the protocol being pretty simple.

[relistan]

Options that I can think of here are:

1. Support existing NRPE daemons with Go implementation of the protocol
2. Support NRPE config translation but with a new implementation (e.g. GRPC or JSON/HTTPS)

I'll take a look at this one to get an eye on how hard item 1 is above.

[dselans]

Nice! I know that the NRPE protocol is fairly simple, but I haven't done any sort of a deep-dive - glad to see exploratory surgery!

[relistan]

Schwing! MIT-licensed Go implementation: https://github.com/envimate/nrpe

[relistan]

Hmm, seems to use OpenSSL and CGO rather than native Go SSL. Might need some work. Looks like NRPE only supports very nasty old SSL and that may be the reason.

[relistan]

NRPE uses ADH (anonymous Diffie Helman) cipher suites which are not supported by Go's TLS package: https://golang.org/src/crypto/tls/cipher_suites.go . Apparently it could be implemented in a fork of the Go TLS lib (e.g. https://groups.google.com/forum/#!topic/golang-nuts/URhpLNZWgL8/discussion) . I suggest that if we decide to support NRPE for legacy integration purposes, we do it in a separate service that can do the nasty CGO and integration with the threaded/heavily locking OpenSSL stuff. We could then call between 9volt and that service with GRPC.

Otherwise, we could:

1. Implement a server that pulls in the NRPE configs and replaces it with something more modern
2. Build a fork of NRPE that can be installed in place of an existing install, but which supports cert auth
3. Suggest that people run NRPE behind something like Stunnel, or a simple Go service
4. Potentially something else?

Thoughts?

[dselans]

Huge bummer about the copher suite not being supported.

Let me ponder about this a bit longer after work. Will update soon.

[dselans]

Optimally it should be the least difficult route for adoption. Requiring folks to update their existing NRPE setups across hosts would be a pain...

Maybe we can build two 9volt's -- 9volt and 9volt-nrpe? That way folks that don't need it can just pull 9corp/9volt and folks who need it, 9volt-nrpe? Utilize build tags as part of the compile/build chain.

The issue that I foresee with having a separate 9volt-nrpe service is that then we have to worry about that being H/A and/or distributed...

cc: @relistan would like your opinion

[relistan]

So I hacked up a little demo of my GRPC idea. https://github.com/relistan/nrpe-grpc

The idea is you just run this on the box that has 9volt and you don't do much but have 9volt expect it to be there. (It's dead simple, so just manage it with S6 or something). Run it in a Docker container, whatever. It does nothing except isolate the OpenSSL libs and calls to NRPE to an external service.

You can build and run the demo and then pass the client the name of the NRPE check to run. It doesn't currently pass params but you get the gist.

Obviously 9volt would call the server directly, not using the CLI client.

[relistan]

To sketch that out a little better, the idea would be a check in 9volt that is described as a "nrpe" check but which just makes grpc calls to the NRPE bridge. The configuration is all in 9volt and each check runs just like any other check. But instead of calling out to the remote node, it connects to the local NRPE bridge which does the actual NRPE call and replies back to 9volt. No coordination or redundancy is required for the bridge service, it relies on 9volt for everything that matters. You run one bridge per node that runs 9volt. Options for keeping it running are:

1. People can manage it themselves
2. 9volt could manage it

The second option wouldn't be that bad since the only thing that needs to happen is for it to actually stay running since it has no config.

[dselans]

Yep @relistan - makes complete sense. I'd like to think about how it'll look from docker-side of things - do we bundle both of the bins in the same image and 9volt execs the nrpe helper (and keeps it running)? Maybe 9volt determines if it should launch the nrpe helper if an env var is set?

Ie. NINEVOLT_ENABLE_NRPE by default is false; if someone wants nrpe, they have to ensure that env var (or flag) is set.

Need to think about what it'll look like on bare-metal as well, turns out not everyone runs their infra on docker 😄

[relistan]

Looks like in mid-Summer 2016 NRPE got support for much better SSL. Not sure what 9volt should support but my guess is most people who would want to switch would be running old-school stuff.