Logging against Elastic Search with X-Pack Security Enabled

[dandrejvv]

It seems as that this plugin doesn't cater for Elastic Search when X-Pack is installed with the Security feature switched on. It seems to use OAuth security. Would it be possible to allow this NLog plugin to allow one to configure it to authenticate against Elastic Search's security feature in order to log to Elastic? Or is there some other feature in Elastic that I'm missing? At the moment I am logging to Elastic without authentication.

[markmcdowell]

I haven't tried it but https://www.elastic.co/guide/en/x-pack/6.0/http-clients.html seems to imply it's just basic authentication.

Have you tried using the requireAuth setting?

[dandrejvv]

Hi @markmcdowell. Thanks for the help. I see now how it works. I haven't noticed those settings in the plugin. I'll try it out and let you know.

[dandrejvv]

Ok, so I've tested it and it is working. When I specify the requireAuth to be true and the username with the password it logs the message on Elastic with credentials. If I omit those, it fails but in the logs (NLog's internal logging) it doesn't say why it failed:

2017-12-08 08:18:09.8820 Error Failed to send log messages to elasticsearch: status=401, message="One or more errors occurred."
2017-12-08 08:18:09.8881 Trace Failed to send log messages to elasticsearch: result=Unsuccessful low level call on POST: /_bulk
2017-12-08 08:18:09.8881 Error Error while sending log messages to elasticsearch: message="One or more errors occurred."

[markmcdowell]

    Ok cool, the status=401 is unauthorised 

    Thanks, Mark

On Fri, Dec 8, 2017 at 6:21 AM +0000, "Dandré" notifications@github.com wrote:

Ok, so I've tested it and it is working.

When I specify the requireAuth to be true and the username with the password it logs the message on Elastic with credentials.

If I omit those, it fails but in the logs (NLog's internal logging) it doesn't say why it failed: 2017-12-08 08:18:09.8820 Error Failed to send log messages to elasticsearch: status=401, message="One or more errors occurred." 2017-12-08 08:18:09.8881 Trace Failed to send log messages to elasticsearch: result=Unsuccessful low level call on POST: /_bulk 2017-12-08 08:18:09.8881 Error Error while sending log messages to elasticsearch: message="One or more errors occurred."

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kcm]

Hey, we tried getting this working with Logstash using HTTPS, but without auth. We're using a self-signed certificate on the LS side, with the CA root added to the NLog-side MMC cert store. No dice. We then added basic auth and enabled requireAuth, with no change or progress. Logging shows no real errors.

Is requireAuth + auth required to support an HTTPS protocol URI? It didn't work either way, but it would be nice to know if HTTPS would work without auth going forward.

We can successfully use cURL to post the bulk data with and without auth, using HTTPS, to the LS endpoint.

For completeness, we're currently using an nginx reverse proxy on the NLog HTTP localhost side to connect via HTTPS to the Logstash endpoint successfully, but we'd like to remove this shim.

Also, if you're using Logstash with the http input plugin, you'll want to use a configuration similar to:

input {
  http {
    codec => es_bulk
    additional_codecs => {}
  }
}

The additional_codecs will prevent the Content-Type: application/json from accidentally applying additional codecs by default, and properly interpret the contents of the bulk post rather than the bulk post itself.