logstash-plugins / logstash-output-http

Apache License 2.0
35 stars 82 forks source link

Add log_http_payload option #116

Closed jsmucr closed 3 years ago

jsmucr commented 4 years ago

This commit adds a new option to disable large HTTP payloads from being logged along with error messages.

Logging payloads that Logstash failed to send can lead to "explosions" in workflows like this:

  1. Logstash is running as systemd service.
  2. System journal is forwarded into Graylog.
  3. Graylog filters input data and separates them into streams using patterns.
  4. One of the streams is then forwarded to Logstash for further processing.
  5. The Logstash instance tries to forward the data to an HTTP endpoind and fails to do so.
  6. Logstash announces an error and includes the HTTP payload.
  7. The error message gets into the journal and back to Graylog where it matches the stream rules again. (=> 2)

This approach is way simpler than trying to fiddle with Log4j2 regex replace in order to get rid of the :body=> log entry.

cla-checker-service[bot] commented 4 years ago

❌ Author of the following commits did not sign a Contributor Agreement: ce75a1db7b650ad3e1bc7194d05a4682e72bb463

Please, read and sign the above mentioned agreement if you want to contribute to this project

jsvd commented 3 years ago

Closing as this is address by #122, where body and headers will only be logged in debug mode