Empty response when Origin, U-A and Referer set.

[codebreach]

Elasticsearch version: 2.3.2

JVM version: 8

Description of the problem including expected versus actual behavior: Certain requests lead to net::ERR_EMPTY_RESPONSE or equivalent curl: (52) Empty reply from server when all Referer, Origin and User-Agent headers are specified.

Steps to reproduce:

1. Start a server with allow_cors: true and allow_cors_origin: "*"
2. Try to use server at IP:9200 (works)
3. Try to use from a different host - fails

It seems something fishy is going on with the header parsing. Combination of Referrer, Origin and User-Agent headers make it fail. Just two of the three headers work as far as I can tell.

Provide logs (if relevant):

CURL attempts:

[madhav:~/Code/es-test] 52 $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'Origin: http://localhost:8000' -H 'Save-Data: on' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' --compressed
curl: (52) Empty reply from server
[madhav:~/Code/es-test] 52 $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'Origin: http://localhost:8000' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' --compressed
curl: (52) Empty reply from server
[madhav:~/Code/es-test] 52 $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'Origin: http://localhost:8000' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
curl: (52) Empty reply from server
[madhav:~/Code/es-test] 52 $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'Origin: http://localhost:8000'
{
  "name" : "Bonita Juarez",
  "cluster_name" : "myesdb",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}
[madhav:~/Code/es-test] $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'Origin: http://localhost:8000'
{
  "name" : "Bonita Juarez",
  "cluster_name" : "myesdb",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}
[madhav:~/Code/es-test] $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'Origin: http://localhost:8000' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
curl: (52) Empty reply from server
[madhav:~/Code/es-test] 52 $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'Origin: http://localhost:8000' -H 'User-Agent: Mozilla/5.0'
curl: (52) Empty reply from server
[madhav:~/Code/es-test] $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Origin: http://localhost:8000' -H 'User-Agent: Mozilla/5.0'
curl: (52) Empty reply from server
[madhav:~/Code/es-test] 52 $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
{
  "name" : "Bonita Juarez",
  "cluster_name" : "myesdb",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}
[madhav:~/Code/es-test] $ curl 'http://REDACTED:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8000/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
{
  "name" : "Bonita Juarez",
  "cluster_name" : "myesdb",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

Relevant (seeming) log message:

[2016-06-07 13:15:21,461][WARN ][http.netty               ] [Bonita Juarez] Caught exception while handling client http traffic, closing connection [id: 0xd2005f5f, /10.188.0.1:14847 => /10.188.0.4:9200]
java.lang.IllegalStateException: cannot send more responses than requests
    at org.jboss.netty.handler.codec.http.HttpContentEncoder.writeRequested(HttpContentEncoder.java:101)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:254)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:292)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:254)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:105)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
    at org.elasticsearch.http.netty.cors.CorsHandler.forbidden(CorsHandler.java:130)
    at org.elasticsearch.http.netty.cors.CorsHandler.messageReceived(CorsHandler.java:84)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

[abeyad]

@codebreach Using a 2.3.2 installation, I set http.cors.enabled to true and http.cors.allow-origin to "*" in elasticsearch.yml. I subsequently ran all of your commands listed, and all of them returned to me a successful response. CORS handling does not take into account the Referrer and as of v2.3.3, the User-Agent is not taken into account either (https://github.com/elastic/elasticsearch/pull/18283).

Could you kindly elaborate on what you mean by trying to use on IP:9200 works but on a different server it fails?

Your stack trace is disconcerting as it seems to indicate the netty channel is responding twice to the same request.

Are you able to reproduce this reliably?

[codebreach]

Regarding the IP: sorry got too lost copying over stack traces

If I talk directly to IP:9200 (using chrome at http://IP:9200/) everything works just fine BUT if I use something like elasticsearch-head running in its own server (basically try to hit the end point IP:9200 from a different origin), all hell breaks lose

I can diagnose more if you have any tips. As I am pretty sure this 'bug' affects a large enough surface area that it should have been reported already, so it could be something else causing it.

FWIW, I am running in kubernetes behind a loadbalancer (on google container engine). Not sure if that can make this blow up. I am going to try to hit the node directly without the LB and see if that changes anything.

[codebreach]

Also it seems I have cors_allow_origin set to localhost and not "*" as reported earlier.

[abeyad]

@codebreach if you are trying to hit the end point from a different origin, and you only have the http.cors.allow-origin set to localhost, then the request should be denied, right?

[codebreach]

Yes I have http.cors.allow-origin set to localhost and have tried curl with Origin and Referer set to localhost with same result. Also I tried to set http.cors.allow-origin to "*" and also ran curl from my laptop; within my cluster (both with and without LB) with same results:

$ kubectl exec curl-deployment-2188210394-dlwy4 -- curl 'http://10.191.246.9:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost/' -H 'Origin: http://localhost/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 52

$ kubectl exec curl-deployment-2188210394-dlwy4 -- curl 'http://10.188.0.6:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost/' -H 'Origin: http://localhost/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 52

[codebreach]

I also played around a bit and found something shady:

This works:

curl 'http://IPADDRESS:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost/' -H 'Origin: http://localhost/' -H 'User-Agent: Mozill'

This doesn't:

curl 'http://IPADDRESS:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost/' -H 'Origin: http://localhost/' -H 'User-Agent: Mozilla'

Note the only difference is 'User-Agent: Mozill' and 'User-Agent: Mozilla' Mozills MozillGIBBRISH and anything that doesn't start with Mozilla works!!?

[abeyad]

Note the only difference is 'User-Agent: Mozill' and 'User-Agent: Mozilla' Mozills MozillGIBBRISH and anything that doesn't start with Mozilla works!!?

That's correct. All browsers send Mozilla followed by something else, hence the check. The User-Agent check is actually completely removed in v2.3.3.

Also I tried to set http.cors.allow-origin to "*" and also ran curl from my laptop; within my cluster (both with and without LB) with same results:

So if you run instead (taking out the Referrer):

$ kubectl exec curl-deployment-2188210394-dlwy4 -- curl 'http://10.188.0.6:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Origin: http://localhost/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'

Then you are saying that works fine for you?

[codebreach]

Without Referer 👎 $ kubectl exec curl-deployment-2188210394-dlwy4 -- curl 'http://10.188.0.6:9200/' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Origin: http://localhost/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' curl: (52) Empty reply from server

[abeyad]

So what is the command that actually works for you, in the exact same setup in which you are running these commands? I ask because you mentioned in the beginning that specifying 2 out of the 3 headers works for you.

[codebreach]

The commands as listed are straight from the terminal. User-Agent seems to be the culprit.

Aside: is there a way to see the config (cors and everything) that the ES server is running with?

[abeyad]

The settings you are probably looking for are:

curl -XGET "localhost:9200/_nodes?pretty"

What do you mean by User-Agent is the culprit? Could you try just setting User-Agent to Mozilla XYZ instead of Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 and see if that makes a difference?

[codebreach]

Thanks - I just double checked, CORS is enabled in config:

       "http" : {
          "compression" : "true",
          "enabled" : "true",
          "cors" : {
            "allow_origin" : "*",
            "enabled" : "true"
          }
        },

HOWEVER there are no CORS headers! I looked at #9031 and #16689 and seems they would get CORS headers when UA was Mozilla and not otherwise, but I am getting an empty response

$ curl -H "User-Agent: Mozilla" -H "Origin: http://example.com" -i http://localhost:9200/
curl: (52) Empty reply from server
$ curl -H "User-Agent: Mozill" -H "Origin: http://example.com" -i http://localhost:9200/
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 309

{
  "name" : "Urthona",
  "cluster_name" : "myesdb",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

[codebreach]

Andd I just wasted your time because of a typo :( allow_origin was specified and not allow-origin

Still might be worth it to look at why that causes the server to start throwing exceptions. But I'm going to close this out. Thanks for all the help investigating.

[abeyad]

@codebreach No problem! Glad it didn't end up being a blocker for you.