search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
103 stars 4.92k forks source link

[Filebeat] HTTP(S) proxy support within module azureeventhub #26328

Open daghemo opened 3 years ago

daghemo commented 3 years ago

The azureeventhub module found on https://github.com/elastic/beats/tree/master/x-pack/filebeat/input/azureeventhub does not support HTTP(S) proxies in order to access Azure Event Hubs. The problem is documented on https://github.com/Azure/azure-event-hubs-go/issues/163. That is, the module uses a Go library from Microsoft, but this library does not support HTTP(S) proxies right now.

Adding proxy support would allow Filebeat and its azureeventhub module to work from behind a corporate HTTP(S) proxy.

elasticmachine commented 3 years ago

Pinging @elastic/integrations (Team:Integrations)

rbr101 commented 3 years ago

yes please, add proxy support so we can use it behind a corporate proxy

narph commented 3 years ago

not sure this is a workaround but might worth trying. Found https://github.com/Azure/azure-event-hubs-go/blob/master/eph/eph.go#L122 in the sdk we are using to collect events. Looking at the code https://github.com/Azure/azure-event-hubs-go/blob/94590f4a150a07d4312ec6027e670c85c3616059/namespace.go#L104 it has the option to use WebSockets but we need to evaluate the full impact of exposing this option. Keep in mind an https proxy might not support WebSockets either.

daghemo commented 3 years ago

Hi @narph and thank you for your contribution. No, your workaround cannot apply here. :( We are required to use HTTPS and to pass all outgoing connection through an HTTPS proxy for Security.

rbr101 commented 3 years ago

This should be implemented to use with the filebeat azure module --> https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-faq

The official Azure SDKs generally use the AMQP protocol for sending and receiving events from Event Hubs. The AMQP-over-WebSockets protocol option runs over port TCP 443 just like the HTTP API, but is otherwise functionally identical with plain AMQP. This option has higher initial connection latency because of extra handshake round trips and slightly more overhead as tradeoff for sharing the HTTPS port. If this mode is selected, TCP port 443 is sufficient for communication. The following options allow selecting the plain AMQP or AMQP WebSockets mode:

We don't mind the extra latency, opening the AMQP port in the corporate firewall is not allowed so we have to go with port 443 using a proxy.

daghemo commented 3 years ago

I just want to clarify that the problem is with https://github.com/Azure/azure-event-hubs-go/, not with Filebeat on itself.

Filebeat uses the above library from Microsoft in other to communicate with Azure Event Hubs. Actually different libraries exists for different languages, from Java to Go. The Java library seems to support HTTP(S) proxies. The Go library no.

In order words... put a thumbs up on https://github.com/Azure/azure-event-hubs-go/issues/163! ;)

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

daghemo commented 1 year ago

+1

Spycol commented 1 year ago

Hi, Is there any plan to implement the proxy configuration somehow? Thanks

botelastic[bot] commented 4 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!