Command `filebeat test output` does not honor proxy_url setting for output.logstash or output.elasticsearch

[fhitchen]

• Version: 7.12.0
• Operating System: Linux X86_64
• Steps to Reproduce:

The output endpoint (logstash or elasticsearch) must only be network accessible via the proxy that you configure.

Configure the output with the proxy_url setting. For example

  hosts: ["hostxys.domain.com:9200"]
  proxy_url: socks5://proxyhost.domain.com:8080

Test the connection using the test output command...

$ ./filebeat test output
elasticsearch: https://hostxyz.domain.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 125.57.74.33
    dial up... ERROR dial tcp 125.57.74.33:9200: i/o timeout

In this example, hostxyz.domain.com is dns resolved locally, not as should be the case by the proxy server, proxyhost.domain.com. The config setting proxy_use_local_resolver defaults to false, but the address is resolved locally. Then filebeat tries to open the network connection directly to the elasticsearch host hostxyz.domain.com and times out as the TCP packets are blocked via the firewall.

$ netstat -t
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      1 myhost.locallan.c:58922 hostxyz.domain.:wap-wsp SYN_SENT

The expected behavior is that the connection to elasticsearch or logstash will be tested via a network connection to the proxy server. This is not the case and it caused a lot of confusion as it appeared that the proxy_url setting was not being picked up. The only way to test the connection was by using the filebeat run command, and that worked perfectly.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ph]

@urso Do you think this fall into the httpcommon effort, if this is the case I've marked it as in progress.