api_request should accept a list of api endpoints for HA

[cwjohnston]

api_request currently constructs the URL for the API end point using the api scope host and port values. As described in sensu/sensu#1299 and sensu-plugins/sensu-plugins-sensu#6, this is problematic when we want handlers to connect to the API on a load balancer whose port differs from the one the API itself is listening on.

Load balancer-specific concerns aside, Sensu Server currently has no way of knowing if the API is unavailable, which means that pipe handlers using this library may fail to execute as expected if the one and only API server they know about (via api config scope) is unavailable for some reason.

I think we can improve this by enhancing sensu-plugin api_request method to support iterating over an array of API endpoints described under a configuration scope that won't require changes to the API server itself, e.g. pointing at a single load balancer:

{ "api": { "endpoints": [ { "host": "192.168.10.11", "port": 45670 } ] } }

or directly at one or more APIs:

{ 
  "api": {
    "endpoints": [ 
      { 
        "host": "192.168.10.9",
        "port": 4567
      },
      { 
        "host": "192.168.10.10",
        "port": 4567
      }
    ] 
  } 
}

[cwjohnston]

We can also try to address #90 and/or #95 in the same change.

[cwjohnston]

In sensu/sensu#1299 @jaykumar-jnpr asks:

what is the expected behavior when one of the API dies in the Array, do we have any health check logic to ignore dead API endpoint? Or just basic timeout with exponential back-off.

I believe that because sensu-plugin invocations are short-lived, i.e. the library is invoked when Sensu forks a process to run a plugin, we'd probably need a short timeout on each request so as to handle issues as quickly as possible. Exponential back-off would likely have major performance effects.

I imagine we would iterate over the array of end points until we found one that handles our request successfully, or we hit the execution timeout for the handler.

[cwjohnston]

I'm still open to feedback on this, and not entirely sure its necessary in light of #134. :)

[cwjohnston]

This api.endpoints spec should probably be adopted/canonized in Sensu Core before we try to create another defacto standard here...

[cwjohnston]

@portertech what do you think about implementing defaults for api.endpoints in sensu-settings for use by extensions and plugins?

[cwjohnston]

API endpoint configuration validation was added to sensu-settings 9.9.0 in https://github.com/sensu/sensu-settings/pull/65, shipped in Sensu Core 0.28

[jaboc]

Why wouldn't you use something like HA proxy infront of your pool of api servers?