Optional response parameters intermittently cause Gavel.js validation to fail

anth0d commented 7 years ago

Describe your problem

API specification in Swagger format - response body contains two properties, one is required, one is optional.

Actual response body contains only the required property, but intermittently Gavel.js validates against a fake response containing both properties, which fails the test intermittently. Gavel should not be including the optional property in the expected result.

The "lorem ipsum" expected response body should not be generated with optional parameters since that is causing unexpected/invalid test failures.

If you can point me to wherever the expected response body is being generated I would be happy to submit a PR to address this issue.

  MyResponse:
    type: object
    required: [foos]
    properties:
      foos:
        type: array
        items:
          type: string
      bars:
        type: array
        items:
          type: string

What command line options do you use?

dredd --details --header "Authorization: Bearer <...>" --config "${DREDDFILE}" --hookfiles "${HOOK_DIR}/${HOOK_BIN}"

What is in your `dredd.yml`?

dry-run: null
hookfiles: null
language: go
sandbox: false
server: null
server-wait: 3
init: false
custom: {}
names: false
only: []
reporter: []
output: []
header: []
sorted: false
user: null
inline-errors: false
details: false
method: []
color: true
level: verbose
timestamp: false
silent: false
path: []
hooks-worker-timeout: 30000
hooks-worker-connect-timeout: 3000
hooks-worker-connect-retry: 1000
hooks-worker-after-connect-wait: 500
hooks-worker-term-timeout: 3000
hooks-worker-term-retry: 500
hooks-worker-handler-host: 127.0.0.1
hooks-worker-handler-port: 61321
config: ./dredd.yml
blueprint: ['swagger/swagger.yml']
endpoint: 'http://0.0.0.0:8080'

What's your `dredd --version` output?

dredd v4.4.0 (Darwin 15.6.0; x64)

Does `dredd --level=debug` uncover something?

Only that the validator is Gavel.js

Can you send us failing test in a Pull Request?

Perhaps, if my initial description doesn't convince you there is an issue.

honzajavorek commented 7 years ago

This is a behavior of the Swagger Fury adapter. I personally do not like the behavior exactly for the reasons you wrote - in the context of Dredd, randomly generating or not generating optional properties makes the tests random as well.

Under the hood, the adapter uses json-schema-faker and IIRC previously I had some discussions on this topic with the @apiaryio/adt, but it always ended with "this is how the faker does the generating".

At the same time, it is strange to me Gavel gives you different results according to what's generated, because it should still validate according to the same schema from which the faked body comes from 🤔 So with or without optional properties, the bodies should pass the schema.

honzajavorek commented 7 years ago

@antkazam Could you share the whole API description? Or a minimal anonymized version of it, which still reproduces the behavior?

honzajavorek commented 7 years ago

@antkazam Any updates on this?

anth0d commented 7 years ago

Sorry about the delay, I was offline for a couple weeks there.

I have been trying to reproduce this issue but I'm not able to with the latest versions. I think we can close this for now.

apiaryio / dredd