search

Azure / diagnostics-eventflow

Microsoft Diagnostics EventFlow
MIT License
305 stars 97 forks source link

Need to set proxy for ElasticSearch output #394

Closed srothkin closed 3 years ago

srothkin commented 3 years ago

Our on-prem environment requires use of a proxy to get to our AWS-hosted Elastic server. By default, NEST (at least under .Net Framework 4.8) doesn't use the system default proxy configuration. And EventFlow Elastic output doesn't expose the underlying NEST proxy configuration settings.

Suggestion - add the various proxy settings in https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/configuration-options.html to EventFlow's Elastic output configuration and use those when creating the Elastic client.

karolz-ms commented 3 years ago

@srothkin thank you for the suggestion. Looks like the API surface for the ES connection options is pretty large. Also it is likely it will be changing from one ES version to the next. For these reasons I am reluctant to attempt to reflect ES connection options into EventFlow configuration schema.

Instead I am thinking we could provide a callback that would allow Elasticsearch output user to customize the ConnectionOptions object that is used by the output to connect to Elasticsearch. The only drawback is that this capability would be available via code. In other words, ES output, together with the rest of EventFlow pipeline would have to be created completely programmatically. At least that is the case today with the current diagnostic pipeline design. Let us know if this would be an acceptable solution.

srothkin commented 3 years ago

No, that's not acceptable. I'd like it to be kept simple - an optional proxy URL in the Elastic Output config JSON - if set, the Elastic Output adds that to the configuration object before creating the Elastic Client. I don't really expect that the proxy url option in the underlying NEST config is going to change from version to version unless they have a larger change that would break other things as well.

karolz-ms commented 3 years ago

OK, sorry, I misunderstood your suggestion--thought you were asking to support all ES connection configuration properties.

Can you confirm that setting the proxy address, username, and password will be sufficient for your use case?

srothkin commented 3 years ago

Already tested/verified that with Internet Explorer proxy settings (Windows Server 2012 R2).

karolz-ms commented 3 years ago

@srothkin Elasticsearch output package 2.7.4 has been published to NuGet. Could you give it a test and report if it solves this issue?

srothkin commented 3 years ago

It does, thank you.