gzip encoding not decoded properly in dredd

[yuriabaevzooz]

Describe your problem

My requests to the service being sent with Accept-Encoding: gzip header. which means the response results in a gzip encoding format, which in dredd are shown as question marks

when I dont send requests with Accept-Encoding: gzip header dredd decodes the result as expected.

What command line options do you use?

$ dredd 

What is in your dredd.yml?

reporter: apiary
custom:
  apiaryApiKey: *****
  apiaryApiName: *****
dry-run: null
hookfiles: ./hook.js
language: nodejs
sandbox: false
server: null
server-wait: 3
init: false
names: false
only: []
output: []
header: []
sorted: false
user: null
inline-errors: false
details: false
method: []
color: true
level: info
timestamp: false
silent: false
path: []
hooks-worker-timeout: 5000
hooks-worker-connect-timeout: 1500
hooks-worker-connect-retry: 500
hooks-worker-after-connect-wait: 100
hooks-worker-term-timeout: 5000
hooks-worker-term-retry: 500
hooks-worker-handler-host: localhost
hooks-worker-handler-port: 61321
config: ./dredd.yml
blueprint: ./api-description.yml
endpoint: "*********"

What's your dredd --version output?

2.2.5

Does dredd --level=debug uncover something?

Can you send us failing test in a Pull Request?

i

fail: body: Real body "Content-Type" header is "application/json;charset=UTF-8" but body is not a parsable JSON. Parse error on line 1: �� �0 �U ^ Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined'

[kylef]

I've attempted to solve this using a dredd hook to decode gzip, but unfortunately since dredd deals with strings and not buffer this doesn't appear to be possible.

@honzajavorek perhaps dredd could handle this, or provide the hooks to get hold of the raw buffer of data instead of providing it as a string (which could loose information).

Here's what I tried:

var zlib = require('zlib');
var hooks = require('hooks');

hooks.beforeEachValidation(function (transaction, done) {
  if (transaction.real.headers['content-encoding'] === 'gzip') {
    zlib.gunzip(transaction.real.body, function(err, buffer) {
      transaction.real.body = buffer.toString('utf-8');
      done();
    });
  }
});

[honzajavorek]

@yuriabaevzooz Thank you for filing this and @kylef thank you for trying to find a solution. It would be good to revisit tests, docs and examples on how Dredd handles binary requests/responses. I think this is kind of related and could go together with it.

It would be nice if we could first think about what would be the best expected solution to this. I don't think having buffer in transaction.real.body is a good API, because you cannot transfer buffer like that to other than JS hooks (the interface between Dredd and non-JS hooks is a socket).

• Should Dredd decode/encode gzipped requests/responses?
• If so, how do you test if the response was gzipped or not?

[vejja]

If so, how do you test if the response was gzipped or not?

If the server sent the Content-Encoding: gzip header

[kylef]

I don't think having buffer in transaction.real.body is a good API, because you cannot transfer buffer like that to other than JS hooks (the interface between Dredd and non-JS hooks is a socket).

That is not correct, sockets send bytes over, strings are being converted from a string representation into unicode in the current implementation in Dredd.

Nodes socket.write is actually encoding strings as utf8 by default (https://nodejs.org/api/net.html#net_socket_write_data_encoding_callback) and in Python hook this is decoded from bytes into a String (https://github.com/apiaryio/dredd-hooks-python/blob/master/dredd_hooks/dredd.py#L77).

[w-vi]

I think that content-encoding: gzip could be handled by Dredd itself. User could just pass a flag saying if content-encoding: gzip then unzip and it can be expressed somehow in the reporter that given response was encoded. Or maybe the other way around as I think that for most folks the compression is almost invisible (handled by nginx at the front) and therefore it could work out of box in dredd too. I actually feel that gzip encoding is not part of the API but more part of the transportation layer so it should work without any effort on user's side.

[honzajavorek]

I agree with @w-vi that Dredd should take care of it. I think for most people gzipping is something which happens automatically. And you should be still able to test it, thanks to the header, as @vejja points out.

[henrihs]

A bit late to the party here, but this is a working solution with js hooks:

hooks.before("<transactionName>", (transaction) => {
    transaction.request.headers["Accept-Encoding"] = "gzip";
  });

hooks.beforeValidation("<transactionName>", (transaction) => {
    const base64EncodedString = transaction.real.body;
    const buffer = Buffer.from(base64EncodedString, "base64");
    const decompressed = zlib.gunzipSync(buffer);
    transaction.real.body = decompressed.toString();
  });