add response body logging to non 2xx responses

jsvd commented 6 months ago

Log the response body ~~and the event (in JSON format)~~ when the endpoint returns a non 2xx response.

Before, the logging entry included :event => event, but the event wasn't being displayed at all, it was just an stringified object reference. ~~Given the potential size of the event or response size, both of these are only logged in debug mode.~~ This PR makes logging more consistent with the retryable error logging.

jsvd commented 5 months ago

@andsel I just simplified the PR for now to add the logging of the body. I also removed the event logging since it was broken already. it can be added in a separate PR.

andsel commented 5 months ago

Tested with simple failing pipeline:

input {
  generator {
    message => '{"greetings": "Hello world!"}'
    count => 1
    codec => json
  }
}

output {
  http {
    url => "https://google.it/sink"
    http_method => "post"
  }
}

Log before

[2024-06-03T15:27:23,040][ERROR][logstash.outputs.http    ][main][http_out] Encountered non-2xx HTTP code 404 
{:response_code=>404, :url=>"https://google.it/sink", :event=>#<LogStash::Event:0x6992dbab>}

Log after

2024-06-03T15:35:30,184][ERROR][logstash.outputs.http    ][main][http_out] Encountered non-2xx HTTP code 
{:code=>404, :body=>"<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  
<title>Error 404 (Not Found)!!1</title>\n  
<style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n  
</style>\n  
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n  
<p><b>404.</b> <ins>That’s an error.</ins>\n 
 <p>The requested URL <code>/sink</code> was not found on this server.  <ins>That’s all we know.</ins>\n", :url=>"https://google.it/sink"
}

logstash-plugins / logstash-output-http

add response body logging to non 2xx responses #142

Log before

Log after