unable to unzip gzipped files in custom app using Passenger 5

[littleforest]

We have a setup where we are using ActiveSupport::Gzip to compress plain text data and send it to a custom app using the send_data method. The app uses the standard "deflate" compression. We have a working production site using Ruby 1.9.3, Rails 3.2 and Passenger 3.0.19 on Ubuntu 14.04.2. LTS.

We upgraded to Ruby 2.1.5 and the latest stable Passenger 4 version, but had issues with the app receiving the contents of the send_data method, we believe due to the chunking bug that was fixed in version 5. After we upgraded to version 5.0.7, the app can now receive the contents, but cannot unzip it. We believe this is an issue with Passenger, as our configuration on development using Thin as the webserver functions with no issues.

The app uses the standard deflate compression method, per these specs. We have tried using the Zlib::GzipWriter method and passing different compression levels, but nothing works. Is it possible that Passenger is mangling something along the way, causing the app to be unable to read the file? We can hit the same URL that the app is requesting in a browser and have no issues unzipping the file.

A working production file is posted here, and the failing staging file is posted here.

Thanks for any help you can provide. We are at a complete loss.

[FooBarWidget]

What headers does your app send? Does it send "Content-Encoding: deflate"? I believe "deflate" is very different from "gzip", so if you do then you should send "Content-Encoding: gzip" instead.

[littleforest]

The app sends very minimal headers:

GET /api/signed_urls?access_token=abcdefg&product_name=Revealing+Philosophy&user_id=25285&timestamp=1431105514 HTTP/1.1
Host: staging.interactivegradebook.com
User-Agent: LiveCode (MacOS)

We don't believe the content-encoding header is particularly relevant, because the app is just expecting raw binary data to be returned.

The app is expecting a specific byte sequence in the file header of the returned gzip file. On production (the working site, Passeger 3), the byte sequence returned is:

31 139 8 0 48 225 75 85 0 3

which is expected. On our staging, Passenger 5 site, the byte sequence is:

49 57 13 10 31 139 8 0 225 192

The app is expecting an initial four header bytes of 31 139 8 0. Is Passenger altering the byte steam in some way during transmission?

[FooBarWidget]

Passenger shouldn't alter the byte stream, except to add a chunked transfer-encoding in some cases. Are there any header differences between Passenger 3 and Passenger 5?

[littleforest]

Here is what the app is receiving from our Passenger 3 server:

HTTP/1.1 200 
Cache-Control: private
Content-Disposition: attachment; filename="scores_user_id_25394.gzip"
Content-Transfer-Encoding: binary
Content-Type: application/octet-stream
Date: Wed, 13 May 2015 19:07:49 GMT
ETag: "2444c97db3a71637d9174b950a05ae0b"
Server: Apache/2.4.4 (Unix) OpenSSL/1.0.1f Phusion_Passenger/3.0.19
Status: 200
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.19
X-Rack-Cache: miss
X-Request-Id: 6348cf363ff082cf3f353d72954c0cf2
X-Runtime: 1.269430
X-UA-Compatible: IE=Edge,chrome=1
Content-Length: 333
Connection: keep-alive

Here is what is being received by the app from our Passenger 5 server:

HTTP/1.1 200 OK
Date: Wed, 13 May 2015 19:11:21 GMT
Server: Apache/2.4.4 (Unix) OpenSSL/1.0.1f Phusion_Passenger/5.0.7
x-rack-cache: miss
content-disposition: attachment; filename="scores_user_id_25285.gz"
cache-control: private
content-transfer-encoding: binary
x-ua-compatible: IE=Edge,chrome=1
x-runtime: 1.002583
x-request-id: ac573f09ed9913a8eefd7e7cc9702616
Connection: close
X-Powered-By: Phusion Passenger 5.0.7
etag: "72cfdcdf93544fdd0127e90e84bb849c"
Status: 200 OK
Transfer-Encoding: chunked
Content-Type: application/octet-stream

We found that if the server sends the data as ASCII rather than binary then the app can unzip the data with no issues.

[FooBarWidget]

The culprit may not necessarily be Passenger. It may be Apache for some reason.

Can you test your app in Passenger Standalone, using the 'builtin' engine? That will skip all the Apache/Nginx layers, leaving you with Passenger's core only. You can start your app like this:

passenger start --engine=builtin

Does that trigger the issue too?

[OnixGH]

@littleforest I see what's going on. You are parsing the raw response body without taking into account the response headers, which state that the response body is chunked.

What you are seeing is an extra "49 57 13 10" before your application's expected response "31 139 8 0 225 192", which is a chunking header. Chunking headers specify the length of the data data follows and a line break (translated to ASCII in your example the extra header is "19", meaning the content length that follows is 19 bytes).

All of this is not related to Passenger (receiver is responsible for dechunking).

[littleforest]

Thank you both so much for the help. We are passing the Content-Length header now and the app is able to unzip the file with no problem.