search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.41k stars 1.07k forks source link

400 Bad Request Deleting Widgets when behind Google Loadbalencer #3053

Closed GEverding closed 7 years ago

GEverding commented 8 years ago

We have graylog2 hosted on google compute engine behind their l7 load balancer. We are getting 400 badrequests when we try to delete widgets from a dash board. I haven't looked through you code but this could be caused by how google handles delete requests (http://stackoverflow.com/questions/36135180/google-compute-load-balancer-throws-400-bad-request-on-delete). Basically googles loadbalencer throws a 400 If a Delete request has a body.

Expected Behavior

Delete widget gets removed

Current Behavior

Receive 400 Bad Request when deleting widget from dash board.

Possible Solution

http://stackoverflow.com/questions/36135180/google-compute-load-balancer-throws-400-bad-request-on-delete

Steps to Reproduce (for bugs)

Context

Any one running on google with a L7 loadbalencer can't delete streams, widgets, etc.

Your Environment

Google Compute Engine with a http loadbalencer

kroepke commented 8 years ago

@GEverding Hi, to verify this one, could you please provide the URL (at least the path of it) that triggers this behavior? Preferably the entire request from the developer tools of your browser. The frontend code shouldn't send a HTTP body for this call, at least it isn't apparent.

Thanks!

aduewel commented 8 years ago

@kroepke @GEverding :

I tried to setup graylog behind a google loadbalancer, too. I experience the same issues. In the google stackdriver logs is written: "response_sent_by_backend" and I see a stacktrace if graylog is started in debug mode.

~~Is it possible that this is related to the X-Forwarded-For Header which contains multiple IP-Addresses, because the stacktrace mention something about error during parsing request headers? https://cloud.google.com/compute/docs/load-balancing/http/ : X-Forwarded-For: <client IP(s)>, <global forwarding rule external IP>~~ I wasn't able to reproduce the issue by setting the header manually and accessing the server directly.

Request headers (browser):

DELETE /api/system/notifications/multi_master HTTP/1.1
Host: xxx.eu
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Authorization: Basic xxx=
Origin: http://xxx.eu
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
Content-Type: application/json
Accept: application/json
X-Requested-With: XMLHttpRequest
Referer: http://xxx.eu/system/overview
Accept-Encoding: gzip, deflate, sdch
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4

Request headers (server side): tcpdump

Response headers (browser):

HTTP/1.1 400 Bad Request
Date: Mon, 14 Nov 2016 15:33:56 GMT
Content-Length: 0
Via: 1.1 google
Content-Type: text/html; charset=UTF-8

Stacktrace (started graylog2 in debug mode):

2016-11-14T15:33:56.692Z DEBUG [HttpCodecFilter] Error parsing HTTP header
java.lang.IllegalStateException: Bad HTTP headers
    at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:537) [graylog.jar:?]
    at org.glassfish.grizzly.http.HttpServerFilter.handleRead(HttpServerFilter.java:334) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111) [graylog.jar:?]
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536) [graylog.jar:?]
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [graylog.jar:?]
    at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) [graylog.jar:?]
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) [graylog.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) [graylog.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
aduewel commented 8 years ago

@GEverding A workaround would be: Install an additional reverse proxy in front of graylog, I have successfully tested this with apache2 and mod_proxy:

ProxyPass / http://localhost:9000/
ProxyPassReverse / http://localhost:9000/
GEverding commented 8 years ago

Why does that work?

Garrett

On Tue, Nov 15, 2016 at 3:58 AM -0500, "André Düwel" notifications@github.com wrote:

@GEverding A workaround would be:

Install an additional reverse proxy in front of graylog, I have tested this with apache2 and mod_proxy:

ProxyPass / http://localhost:9000/ ProxyPassReverse / http://localhost:9000/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

aduewel commented 8 years ago

Not sure, but something has to be different with the request. Hope the tcpdump above helps a developer to investigate this issue. I believe this is not a graylog issue, but a glassfish/grizzly one.

GEverding commented 7 years ago

Thanks @aduewel. I was just going to throw in a L4 loadbalancer

nathantsoi commented 7 years ago

im seeing the same issue, but behind a different, custom reverse proxy. long cookie strings seem to trigger the behavior. somewhere between 512 and 1024 bytes int he cookie header and i get a 400

nathantsoi commented 7 years ago

figured out how to enable debugging log output and now i see: 

2017-07-06T14:30:41.781-07:00 DEBUG [HttpCodecFilter] Error parsing HTTP header
java.lang.IllegalStateException: HTTP packet header is too large
    at org.glassfish.grizzly.http.HttpCodecFilter$HeaderParsingState.checkOverflow(HttpCodecFilter.java:2020) ~[graylog.jar:?]
    at org.glassfish.grizzly.http.HttpCodecFilter.decodeHttpPacketFromBytes(HttpCodecFilter.java:728) ~[graylog.jar:?]
    at org.glassfish.grizzly.http.HttpCodecFilter.decodeHttpPacket(HttpCodecFilter.java:697) ~[graylog.jar:?]
    at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:545) [graylog.jar:?]
    at org.glassfish.grizzly.http.HttpServerFilter.handleRead(HttpServerFilter.java:334) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) [graylog.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) [graylog.jar:?]
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [graylog.jar:?]
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) [graylog.jar:?]
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [graylog.jar:?]
    at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) [graylog.jar:?]
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) [graylog.jar:?]
    at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) [graylog.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) [graylog.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
nathantsoi commented 7 years ago

ok, fixed. doubling rest_max_initial_line_length = 4096 in the config worked for me  

jalogisch commented 7 years ago

as this looks solved, i'll close this.

If you do not feel, please reopen the issue.