Gavel.js error invalid media type causes Dredd crashing

ywett02 commented 7 years ago

Describe your problem

After test fails, dredd stops with following error:

TypeError: this.emitError is not a function
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:638:14
    at proxy (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/validate.js:16:12)
    at Object.isValid (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/validate.js:32:10)
    at TransactionRunner.validateTransaction (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:635:18)
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:592:28
    at TransactionRunner.runHooksForData (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:191:14)
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:588:26
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:188:16
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1012:9
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:359:16
    at replenish (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:876:25)
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:885:9
    at _asyncMap (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1005:5)
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1091:16
    at timeLimit (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:4785:3)
    at Object.timesSeries (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:917:16)
    at TransactionRunner.runHooksForData (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:187:20)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:583:24)
    at IncomingMessage.g (events.js:260:16)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at nextTickCallbackWith2Args (node.js:441:9)
    at process._tickCallback (node.js:355:17)

What command line options do you use?

$ dredd

What is in your `dredd.yml`?

reporter: apiary
custom:
  apiaryApiKey: ca38493e5954900a1b39071a6768e935
  apiaryApiName: clubspire
dry-run: null
hookfiles: null
language: nodejs
sandbox: false
server: null
server-wait: 3
init: false
names: false
only: []
output: []
header: []
sorted: false
user: null
inline-errors: false
details: false
method: []
color: true
level: info
timestamp: false
silent: false
path: []
hooks-worker-timeout: 5000
hooks-worker-connect-timeout: 1500
hooks-worker-connect-retry: 500
hooks-worker-after-connect-wait: 100
hooks-worker-term-timeout: 5000
hooks-worker-term-retry: 500
hooks-worker-handler-host: localhost
hooks-worker-handler-port: 61321
config: ./dredd.yml
blueprint: apiary.apib
endpoint: 'https://api.clubspire.com'

What's your `dredd --version` output?

dredd v2.2.5 (Linux 4.4.0-64-generic; x64)

Does `dredd --level=debug` uncover something?

verbose: Handling HTTP response from tested server
debug: Response from tested server was recieved
verbose: Running 'beforeEachValidation' hooks
debug: Running hooks...
verbose: Running 'beforeValidation' hooks
verbose: Validating HTTP transaction by Gavel.js
debug: Determining whether HTTP transaction is valid (getting boolean verdict)
debug: Gavel.js validation errored: TypeError: invalid media type
    at splitType (/usr/local/lib/node_modules/dredd/node_modules/media-typer/index.js:249:11)
    at Object.parse (/usr/local/lib/node_modules/dredd/node_modules/media-typer/index.js:160:13)
    at HttpResponse.Validatable.isJsonContentType (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/mixins/validatable-http-message.js:355:27)
    at HttpResponse.Validatable.setBodyExpectedType (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/mixins/validatable-http-message.js:227:16)
    at HttpResponse.Validatable.validateBody (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/mixins/validatable-http-message.js:159:10)
    at HttpResponse.Validatable.validate (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/mixins/validatable-http-message.js:24:12)
    at HttpResponse.Validatable.isValid (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/mixins/validatable-http-message.js:48:12)
    at proxy (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/validate.js:13:39)
    at Object.isValid (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/validate.js:32:10)
    at TransactionRunner.validateTransaction (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:635:18)
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:592:28
    at TransactionRunner.runHooksForData (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:191:14)
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:588:26
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:188:16
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1012:9
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:359:16
    at replenish (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:876:25)
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:885:9
    at _asyncMap (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1005:5)
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1091:16
    at timeLimit (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:4785:3)
    at Object.timesSeries (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:917:16)
    at TransactionRunner.runHooksForData (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:187:20)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:583:24)
    at IncomingMessage.g (events.js:260:16)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at nextTickCallbackWith2Args (node.js:441:9)
    at process._tickCallback (node.js:355:17)

/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:638
        this.emitError(isValidError, test);
             ^

TypeError: this.emitError is not a function
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:638:14
    at proxy (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/validate.js:16:12)
    at Object.isValid (/usr/local/lib/node_modules/dredd/node_modules/gavel/lib/validate.js:32:10)
    at TransactionRunner.validateTransaction (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:635:18)
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:592:28
    at TransactionRunner.runHooksForData (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:191:14)
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:588:26
    at /usr/local/lib/node_modules/dredd/lib/transaction-runner.js:188:16
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1012:9
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:359:16
    at replenish (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:876:25)
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:885:9
    at _asyncMap (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1005:5)
    at /usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:1091:16
    at timeLimit (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:4785:3)
    at Object.timesSeries (/usr/local/lib/node_modules/dredd/node_modules/async/dist/async.js:917:16)
    at TransactionRunner.runHooksForData (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:187:20)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:583:24)
    at IncomingMessage.g (events.js:260:16)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at nextTickCallbackWith2Args (node.js:441:9)
    at process._tickCallback (node.js:355:17)

honzajavorek commented 7 years ago

@ywett02 Thanks! This looks like bug for sure. First, there's the Gavel.js error:

Gavel.js validation errored: TypeError: invalid media type

It's probably caused by something in your apiary.apib file. Is it possible for you to disclose contents of the file or at least a relevant part of it, anonymized?

When the Gavel.js error happens, Dredd isn't able to recover, gracefully report the error and abort testing, but crashes on

/usr/local/lib/node_modules/dredd/lib/transaction-runner.js:638
        this.emitError(isValidError, test);
             ^

TypeError: this.emitError is not a function

That's definitely a bug.

ywett02 commented 7 years ago

@honzajavorek I sent you email with details of our apiary.apib file.

honzajavorek commented 7 years ago

I was able to reproduce the problem with given API Blueprint. I've ran the latest Dredd (3.0.0) like this:

dredd apiary.apib http://google.com --level=silly

I've got the same output as @ywett02 reported above. In the output, following part is the most important:

verbose: Processing transaction #26: Instructors > Instructor photo > Get photo of instructor
verbose: Running 'beforeEach' hooks
debug: Running hooks...
verbose: Running 'before' hooks
debug: Emitting to reporters: test start
verbose: About to perform an HTTP request to tested server
verbose: Handling HTTP response from tested server
silly: Recieving some data from tested server
silly: Recieving some data from tested server
silly: Recieving some data from tested server
debug: Response from tested server was recieved
verbose: Running 'beforeEachValidation' hooks
debug: Running hooks...
verbose: Running 'beforeValidation' hooks
verbose: Validating HTTP transaction by Gavel.js
debug: Determining whether HTTP transaction is valid (getting boolean verdict)
debug: Gavel.js validation errored: TypeError: invalid media type

From the first line it's possible to see the problem is in the Instructors > Instructor photo > Get photo of instructor transaction. The relevant part of the API Blueprint looks like this:

### Get photo of instructor [GET]

Returns photo of instructor of given **id**. Real image data is returned, usually in jpeg format. Additional request attributes can be used to retrieve the image with specific width and height.

Default (phantom) picture is returned in case the instructor has no photo set.

+ Request (application/json)
    + Headers

            Authorization: Bearer xyz

+ Response 200 (image)

The output from Gavel isn't very helpful (it doesn't mention the invalid media type value), but from the API Blueprint it's possible to see the input: image. However, image isn't a valid media type. If you want to test this endpoint, you may need to change the API Blueprint to something like this:

+ Request (application/json)
    + Headers

            Authorization: Bearer xyz

+ Response 200 (image/png)
+ Response 200 (image/gif)
+ Response 200 (image/jpeg)

Basically, you want to list all possible image formats your API supports in the response and which user of your API can expect.

The problem is, Dredd has some problems with testing binary endpoints: https://github.com/apiaryio/dredd/issues/87 So you may want to completely avoid testing this endpoint at the moment. To do that, you can use hooks to skip this particular transaction.

To sum it up, you should:

use specific media types to avoid Dredd crashing
use hooks to deal with the fact Dredd currently doesn't support testing image responses very well

I should:

file an issue (improvement) that it would be nice if Gavel printed also the actual media type when it's invalid
file an issue (bug) that Dredd crashes if Gavel crashes like this, instead of gracefully aborting

honzajavorek commented 7 years ago

I filed https://github.com/apiaryio/dredd/issues/731 and https://github.com/apiaryio/gavel.js/issues/88 as follow-ups. @ywett02 Thank you very much for filing this issue and drawing attention to these problems! 👍 I hope this helped!

apiaryio / dredd