Problem behind AWS ELB

[forficate]

Just trialing rapidoid and have an issue deploying on AWS.

This seems to work ok accessing infront of an ELB:

On.get("/healthcheck").plain("ok");

$ curl -iv https://...../healthcheck
* Connected to *** (***) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: ***
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /healthcheck HTTP/1.1
> Host: ***
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8  
Content-Type: text/plain; charset=utf-8
< Date: Wed, 20 Apr 2016 07:49:44 GMT
Date: Wed, 20 Apr 2016 07:49:44 GMT
< Server: Rapidoid
Server: Rapidoid
< Content-Length: 2
Content-Length: 2
< Connection: keep-alive
Connection: keep-alive

<
* Connection #0 to host *** left intact
ok%

This fails with a 502 Bad Gateway accessing infront of an ELB which suggests something is wrong with the response. Running locally the request works.

On.get("/echo").json((Req req) -> req.params());

The response from http://localhost:8080/echo?a=b is below. I can't see any issues looking which would cause an AWS ELB to return 502, headers look correct etc?

$ curl -iv http://localhost:8080/echo\?a\=b
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /echo?a=b HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK 
HTTP/1.1 200 OK
< Connection: close
Connection: close
< Server: Rapidoid
Server: Rapidoid
< Date: Wed, 20 Apr 2016 07:56:14 GMT
Date: Wed, 20 Apr 2016 07:56:14 GMT
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Content-Length: 9
Content-Length: 9

<
* Closing connection 0
{"a":"b"}%

[nmihajlovski]

Looks like the same problem was reported by another user:

Seems like when you create a get resource using rapidoid-web, the content-length is always padded with space infront which cases some some proxy servers to return a 502, anyone know a way around this?

Given the situation you both described, the Content-Length header looks like this: Content-Length:          9

This is a valid HTTP header, but it seems that ELB parses it incorrectly. The leading spaces in the header are provided when the response size is dynamic, to avoid double buffering of the response. When HTTP response chunks become supported in Rapidoid, it won't be necessary to have the leading spaces.

However, in the upcoming v5.1.0 this problem won't happen, unless when writing directly to: OutputStream Resp#out() For that release I replaced the leading spaces with leading zeros, it is worth trying that: Content-Length: 00000000009

As a final note: Nginx, ab, wrk and HttpClient didn't have any problems with the leading spaces/zeros in the Content-Length header.

[forficate]

I did notice the leading content spaces and created a build without them just to check, that didn't seem to work.

Trying leading zeros also doesn't seem to be working. Running from the ec2 instance I can confirm the Content-Length is padded.

curl -iv  localhost:8080/echo?a=b
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /echo?a=b HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Connection: close
Connection: close
* Server Rapidoid is not blacklisted
< Server: Rapidoid
Server: Rapidoid
< Date: Thu, 21 Apr 2016 00:08:41 GMT
Date: Thu, 21 Apr 2016 00:08:41 GMT
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Content-Length: 00000000009
Content-Length: 00000000009

<
* Closing connection 0

In front of the ELB I see

 curl -iv https://***/echo\?a\=b
 *   Trying ***...
 * Connected to *** (***) port 443 (#0)
 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 * Server certificate: *.dev.atlassian.io
 * Server certificate: DigiCert SHA2 Secure Server CA
 * Server certificate: DigiCert Global Root CA
 > GET /echo?a=b HTTP/1.1
 > Host: ***
 > User-Agent: curl/7.43.0
 > Accept: */*
 >
 < HTTP/1.1 502 BAD_GATEWAY
 HTTP/1.1 502 BAD_GATEWAY
 < Content-Length: 0
 Content-Length: 0
 < Connection: keep-alive
 Connection: keep-alive

<
* Connection #0 to host *** left intact

I'll try and investigate more.

[nmihajlovski]

Thanks for your feedback, today I reproduced the problem with AWS ELB and I fixed the bug (5.1.0 will be released soon).

The problem actually wasn't caused by the leading spaces. The Content-Length header value was rendered at incorrect position, so the CR byte of the CR-LF suffix was overwritten, which made this bug hard to notice. Also, the other software (HTTP clients, proxies) is more flexible and didn't complain about the incorrect headers.

[am-td444]

I think I came across the same issue with Marathon-lb (HAProxy) and healthchecks within Mesosphere so will follow up once 5.1.0 is released.

[nmihajlovski]

This has been fixed in the recent release of v5.1.0.