search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
103 stars 4.92k forks source link

Filebeat: Add HAProxy module support for TLS version and cipher log fields #29318

Open martinseener opened 2 years ago

martinseener commented 2 years ago

Describe the enhancement: At the moment, the exported fields for HAProxy Filebeat module provide a wide variety of fields but it is missing the SSL/TLS Version and Cipher fields. We currently use this log-format and parse this in Logstash but we would like to use only filebeat:

log-format: "%ci:%cp\\ [%t]\\ %ft\\ %b/%s\\ %ST\\ %B\\ %CC\\ %CS\\ %tsc\\ %ac/%fc/%bc/%sc/%rc\\ %sq/%bq\\ %hr\\ %hs\\ %%{}{+Q}r\\ %sslv\\ %sslc\\ %ID"

The last entries provide the version and cipher. Our other option is to migrate this format to JSON and let it parse by an filebeat input but I would prefer the module.

Describe a specific use case for the enhancement or feature: In todays world it is important to know which TLS versions are supported and used in order to decide to enable/disable one for specific interfaces to improve security.

legoguy1000 commented 2 years ago

Can u provide sample of what the logs look like?

elasticmachine commented 2 years ago

Pinging @elastic/integrations (Team:Integrations)

martinseener commented 2 years ago

@legoguy1000 Sorry for late response. The current log looks like this

2021-12-15T16:55:18+01:00 host.example.com haproxy[20560]: <remote-ip>:27368 [15/Dec/2021:16:55:17.995] frontendname~ backendname/server1 24/0/0/3/27 304 453 - - ---- 27/1/1/0/0 0/0 {|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36} "GET /image.png HTTP/1.1" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 R-<someID>
legoguy1000 commented 2 years ago

I can take a look at this when I get a chance.

martinseener commented 2 years ago

Awesome. Let me know if and how I can support you here :-)

legoguy1000 commented 2 years ago

@martinseener whats the R-<someID>??

legoguy1000 commented 2 years ago

Initial PR submitted for review. Still need to run data through it.

martinseener commented 2 years ago

@martinseener whats the R-<someID>??

This is an internal ID haproxy generates so we can use this as a tracing id throughout the system. in haproxy we call it request-id, hence the R-….followed by sone useful id parts that makes the request id unique. the apps behind it take this and use it for further internal requests. maybe we could also add this somehow as an arbitrary field??

legoguy1000 commented 2 years ago

@martinseener whats the R-<someID>??

This is an internal ID haproxy generates so we can use this as a tracing id throughout the system. in haproxy we call it request-id, hence the R-….followed by sone useful id parts that makes the request id unique. the apps behind it take this and use it for further internal requests. maybe we could also add this somehow as an arbitrary field??

Ok, sounds easy enough. Can u provide an actual example so I can make a grok pattern?

martinseener commented 2 years ago

@martinseener whats the R-<someID>??

This is an internal ID haproxy generates so we can use this as a tracing id throughout the system. in haproxy we call it request-id, hence the R-….followed by sone useful id parts that makes the request id unique. the apps behind it take this and use it for further internal requests. maybe we could also add this somehow as an arbitrary field??

Ok, sounds easy enough. Can u provide an actual example so I can make a grok pattern?

Yes! Here is the last portion of the example above without redactions

image.png HTTP/1.1" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 R-6AE8-6249A-61BA0FE5-0A011406-2715-5050

The config to generate this is (yaml config template):

unique-id-format: "%%{}{+X}o\\ R-%cp-%rt-%Ts-%fi-%fp-%pid"
unique-id-header: 'Request-id'
martinseener commented 2 years ago

@legoguy1000 hi, how is the state of the integration?

legoguy1000 commented 2 years ago

@martinseener Sorry this kinda fell off my radar. THe issue i kinda ran into is, from what i could find, the default HAproxy output doesn't include these fields (unless u can show me) and the Unique ID can be any format, not just UUID-esq. THis makes it very difficult to add to the the "default" grok pattern.

martinseener commented 2 years ago

@legoguy1000 you're right. So maybe let's leave it to the 2 TLS components which are very helpful for now. I may think about changing our HAProxy logging to JSON format if I need special cases but maybe we can go with the 2 TLS things first as this is "my" priority!?

legoguy1000 commented 2 years ago

That sounds reasonable but even the TLS fields I don't think are default either per https://www.haproxy.com/blog/haproxy-log-customization/ so that would still be something more specific to your implementation. Is this what you're using to get the JSON logs https://medium.com/thirddev/json-logging-in-haproxy-the-right-way-3636297d2d49??

martinseener commented 2 years ago

The TLS log parameters %sslc and %sslv are indeed not "used" by default in either TCP or HTTP log formats BUT they're built-in to HAProxy logging, so anyone who uses this should be able to also see it in the filebeat module. So you should definitely support this. Of course it can be on a different part of the log line but maybe this module can work with that? If not then - honestly - this module only works for environments with log format defaults (where I guess are not so many).

Yeah I would use something like the medium article you posted. I would use your field names and match our JSON format on to it, so in case we switch, we would still be able to search through new and old logs with the same field names.

legoguy1000 commented 2 years ago

Roger. I'll add the TLS fields as optional on the end and call that a day

EtienneBruines commented 2 years ago

I just came across this, searching for support on the option httpslog that HAProxy offers.

https://cbonte.github.io/haproxy-dconv/2.6/configuration.html#8.2.4

This basically adds (among other things) to the log line, e.g.:

my.super.duper.hostname.com/TLSv1.3/TLS_AES_256_GCM_SHA384

Would it make sense to support things like option tcplog / option httplog / option/httpslog ? Or would it make more sense to define the log-format of haproxy in the settings of metricbeat? https://cbonte.github.io/haproxy-dconv/2.6/configuration.html#8.2.6

martinseener commented 2 years ago

The httplog option is definitely something useful, although with older HAProxy the httpslog was not included, only tcplog and httplog. But customlog was there so one could mimic httpslog with customlog. Either way would be useful!

legoguy1000 commented 2 years ago

I updated the HAproxy integration with these new fields and update the docs to show the supported default patterns, elastic/integrations#3678

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

jehof commented 1 year ago

:+1:

botelastic[bot] commented 2 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!