Use with Serilog Sink

[KyleGobel]

Just looking for solutions at the moment.

There is a plugin for serilog (logging framework) for elasticsearch that uses the elasticsearch package for pumping messages to ES, though I don't have a good idea looking through it's code how i could get it to sign the AWS requests.

I'm wondering if you had any ideas for how to possibly use this with serilog.

(ATM considering forking the current sink they have and dropping in this library and exposing the keys part... unless you've happend to do have done this before or know of a better solution)

[bcuff]

It looks like you would probably need to fork the Serilog Elasticsearch Sink so that you can set up the client in a custom way. Maybe you can simply provide a setup overload that lets you inject a custom ElasticLowLevelClient instance or something and submit a PR back to the main line. https://github.com/serilog/serilog-sinks-elasticsearch/blob/dev/src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticsearchSinkState.cs#L84

You'll then need to setup an ElasticLowLevelClient instance like so

            var httpConnection = new AwsHttpConnection("us-east-1");
            var pool = new SingleNodeConnectionPool(new Uri("https://..."));
            var config = new ConnectionConfiguration(pool, httpConnection);
            var client = new ElasticLowLevelClient(config);

[KyleGobel]

Thanks a lot, your post and little snippet helped me figure out how I could integrate this. Turns out they have a ModifyConnectionSettings hook, I just didn't really know how to use it.

If anyone else was curious (this would probably be better off posted in their repo, but oh well). Here's a pretty crude example of how you could get it to work.

  const string esUrl = "https://aws-es-thinger.us-west-1.es.amazonaws.com";
  Log.Logger = new LoggerConfiguration()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(esUrl))
                {
                    ModifyConnectionSettings = conn =>
                    {
                        var httpConnection = new AwsHttpConnection("us-west-1", new StaticCredentialsProvider(
                            new AwsCredentials
                            {
                                // Service User Keys
                                AccessKey = "",
                                SecretKey = ""
                            }));
                        var pool = new SingleNodeConnectionPool(new Uri(esUrl));
                        return new ConnectionConfiguration(pool, httpConnection);
                    }
                })
                .CreateLogger();

Thanks a bunch for making this library! Saved me probably like 80 hours of fiddling with and trying to sign them requests!

[bcuff]

@KyleGobel I'm glad you figured that out.

That code example would be a good addition to the README. Feel free to PR it. I might do it myself if you don't :).

[mariotacke]

How would one provide basic auth credentials for the connection? In my use case, I have basic auth credentials (not AWS security credentials) that work in the browser, but not in the sink configuration.

[bcuff]

@mariotacke I don't think AWS Elasticsearch Service supports basic http authentication. Either the request must be signed with valid AWS credentials or the inbound traffic needs to come from a whitelisted IP address. You can whitelist IP addresses in your elasticsearch domain's access policy.

[mariotacke]

Thanks @bcuff. I've asked my ops guys to give me proper credentials, that's probably the way to go.

[rtrscience]

Thanks a lot, your post and little snippet helped me figure out how I could integrate this. Turns out they have a ModifyConnectionSettings hook, I just didn't really know how to use it.

If anyone else was curious (this would probably be better off posted in their repo, but oh well). Here's a pretty crude example of how you could get it to work.

  const string esUrl = "https://aws-es-thinger.us-west-1.es.amazonaws.com";
  Log.Logger = new LoggerConfiguration()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(esUrl))
                {
                    ModifyConnectionSettings = conn =>
                    {
                        var httpConnection = new AwsHttpConnection("us-west-1", new StaticCredentialsProvider(
                            new AwsCredentials
                            {
                                // Service User Keys
                                AccessKey = "",
                                SecretKey = ""
                            }));
                        var pool = new SingleNodeConnectionPool(new Uri(esUrl));
                        return new ConnectionConfiguration(pool, httpConnection);
                    }
                })
                .CreateLogger();

Thanks a bunch for making this library! Saved me probably like 80 hours of fiddling with and trying to sign them requests!

This works only in .net core with was elastic search. Can you help me on how to handle the same using with standard asp.net

[bcuff]

You're welcome. Thanks for sharing the snippet!

[orhunerdem]

@mariotacke Hi, any updates on this? I'm in a similar situation and searching web for a solution..

[mariotacke]

@orhunerdem, it's been a long time but IIRC, I ended up using AWS credentials instead of basic auth as suggested by @bcuff and @rtrscience .