search

apiaryio / dredd

Language-agnostic HTTP API Testing Tool
https://dredd.org
MIT License
4.2k stars 280 forks source link

Non listed response keys doesn't fail the test #977

Closed ido567 closed 6 years ago

ido567 commented 6 years ago

Describe your problem

I'm not sure if it's a bug, a configuration problem, or by design - Dredd doesn't fail if the response contains additional structure than described in my swagger.

My Swagger: 

swagger: '2.0'
info:
  title: "Apps Service"
  version: "2.2.0"
host: my-domain.com
basePath: /apps/v1
schemes:
  - https
consumes:
  - application/json
produces:
  - application/json

paths:
  '/products':
    get:
      operationId: getProducts
      responses:
        200:
          description: Success
          schema:
            type: array
            items:
              $ref: '#/definitions/Product'
      tags:
        - Products

definitions:
  Product:
    type: object
    properties:
      id:
        type: string
        readOnly: true
        format: uuid
        example: '15df7fce-e58a-447c-ac95-42661fb73670'
      name:
        type: string
        maximum: 25
        example: Docs
      created_at:
        type: string
        readOnly: true
        example: '2017-11-27T11:53:54.076237Z'
      updated_at:
        type: string
        readOnly: true
        example: '2017-11-27T11:53:54.076274Z'
    required:
      - name

The response:

[
  {
    "id": "4835c6de-91ee-471c-9cb4-99941f04f6b8",
    "name": " DOCS",
    "created_at": "2017-11-10T11:36:45.073000Z",
    "updated_at": "2017-11-10T11:36:45.073000Z",
    "additionalField": "test"
  },
  {
    "id": "e31b300f-6793-4d4e-b62a-074a906015e2",
    "name": "DMAC",
    "created_at": "2017-11-10T11:36:45.487000Z",
    "updated_at": "2017-11-10T11:36:45.487000Z",
    "additionalField": "test"
  }
]

Is Dredd designed for that? If so, how can I achieve this behavior by hooks?

What command line options do you use?

$ dredd --config=./CI/dredd/config.yml

What is in your dredd.yml?

blueprint: https://my-swagger.com
endpoint: http://local.com:8000
hookfiles: ./CI/dredd/hooks.js
language: nodejs
server: 'python manage.py runserver'
details: true
reporter: markdown
output: ['CI/dredd/report.md']
dry-run: false
sandbox: false
server-wait: 3
init: false
custom:
  apiaryApiKey: ''
names: false
only: []
header: []
sorted: false
user: null
inline-errors: false
method: []
color: true
level: info
timestamp: false
silent: false
path: []
hooks-worker-timeout: 5000
hooks-worker-connect-timeout: 1500
hooks-worker-connect-retry: 500
hooks-worker-after-connect-wait: 100
hooks-worker-term-timeout: 5000
hooks-worker-term-retry: 500
hooks-worker-handler-host: 127.0.0.1
hooks-worker-handler-port: 61321

What's your dredd --version output?

dredd v5.1.3 (Darwin 17.4.0; x64)

Does dredd --level=debug uncover something?

Not sure, here is the output:

verbose: Loading configuration file: ./CI/dredd/config.yml
info: Configuration './CI/dredd/config.yml' found, ignoring other arguments.
debug: Dredd version: 5.1.3
debug: Node.js version: v6.9.2
debug: Node.js environment: http_parser=2.7.0, node=6.9.2, v8=5.1.281.88, uv=1.9.1, zlib=1.2.8, ares=1.10.1-DEV, icu=57.1, modules=48, openssl=1.0.2j
debug: System version: Darwin 17.4.0 x64
debug: npm version: 4.2.0
debug: Configuration: {"server":"http://local-apps.api.b360-dev.com:8000","options":{"hookfiles":"./CI/dredd/hooks.js","language":"nodejs","server":"python manage.py runserver","details":true,"reporter":"markdown","output":["CI/dredd/report.md"],"dry-run":false,"sandbox":false,"server-wait":3,"init":false,"custom":{"apiaryApiKey":""},"names":false,"only":[],"header":[],"sorted":false,"user":null,"inline-errors":false,"method":[],"color":true,"level":"debug","timestamp":false,"silent":false,"path":["https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml","https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml"],"hooks-worker-timeout":5000,"hooks-worker-connect-timeout":1500,"hooks-worker-connect-retry":500,"hooks-worker-after-connect-wait":100,"hooks-worker-term-timeout":5000,"hooks-worker-term-retry":500,"hooks-worker-handler-host":"127.0.0.1","hooks-worker-handler-port":61321,"_":["https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml"],"config":"./CI/dredd/config.yml","p":["https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml","https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml"]},"custom":{"cwd":"/Users/idob/Code/integrations-service/integrations","argv":["--config=./CI/dredd/config.yml","--level=debug"]}}
verbose: Using 'base' reporter.
verbose: Configuring reporters: [ 'markdown' ] 0=CI/dredd/report.md
verbose: Using 'cli' reporter.
info: File exists at /Users/idob/Code/integrations-service/integrations/CI/dredd/report.md, will be overwritten...
verbose: Using 'markdown' reporter.
verbose: Backend server process specified, starting backend server and then testing
verbose: Using 'python' as a server command, ["manage.py","runserver"] as arguments
info: Starting backend server process with command: python manage.py runserver
info: Waiting 3 seconds for backend server process to start
info: Backend server process exited
verbose: Running Dredd instance.
verbose: Expanding glob patterns.
verbose: Reading API description files.
verbose: Downloading remote file: https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml
verbose: Parsing API description files and compiling a list of HTTP transactions to test.
verbose: Compiling HTTP transactions from API description file: https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml
warn: Parser warning in file 'https://api.swaggerhub.com/apis/360-apps/2.2.0/swagger.yaml': Unable to generate application/json example message body out of JSON Schema on line 21
verbose: Starting reporters and waiting until all of them are ready.
info: Beginning Dredd testing...
verbose: Starting transaction runner.
verbose: Sorting HTTP transactions
verbose: Configuring HTTP transactions
verbose: Reading hook files and registering hooks
info: Found Hookfiles: 0=/Users/idob/Code/integrations-service/integrations/CI/dredd/hooks.js
verbose: Executing HTTP transactions
verbose: Running 'beforeAll' hooks
verbose: Processing transaction #1: Products > /apps/v1/products > GET > 200 > application/json
verbose: Running 'beforeEach' hooks
debug: Running hooks...
verbose: Running 'before' hooks
debug: Emitting to reporters: test start
verbose: About to perform an HTTP request to the server under test: GET http://local-apps.api.b360-dev.com:8000/apps/v1/products
verbose: Handling HTTP response from tested server
verbose: Running 'beforeEachValidation' hooks
debug: Running hooks...
verbose: Running 'beforeValidation' hooks
verbose: Validating HTTP transaction by Gavel.js
debug: Determining whether HTTP transaction is valid (getting boolean verdict)
debug: Validating HTTP transaction (getting verbose validation result)
verbose: Running 'afterEach' hooks
verbose: Running 'after' hooks
debug: Evaluating results of transaction execution #1: Products > /apps/v1/products > GET > 200 > application/json
debug: Emitting to reporters: test pass
pass: GET (200) /apps/v1/products duration: 84ms
debug: Could not stringify: 
request: 
method: GET
uri: /apps/v1/products
headers: 
    Content-Type: application/json
    Accept: application/json
    User-Agent: Dredd/5.1.3 (Darwin 17.4.0; x64)
    X-Ads-Token-Data: {"access_token":{"userid": "JGGPJFS3G7BR"}}
    Content-Length: 0

body: 

expected: 
headers: 
    Content-Type: application/json

statusCode: 200
bodySchema: {"type":"array","items":{"type":"object","properties":{"id":{"type":"string","readOnly":true,"format":"uuid","example":"15df7fce-e58a-447c-ac95-42661fb73670"},"name":{"type":"string","maximum":25,"example":"Docs"},"created_at":{"type":"string","readOnly":true,"example":"2017-11-27T11:53:54.076237Z"},"updated_at":{"type":"string","readOnly":true,"example":"2017-11-27T11:53:54.076274Z"}},"required":["id","name","created_at","updated_at"]}}

actual: 
statusCode: 200
headers: 
    date: Sun, 25 Feb 2018 10:28:44 GMT
    server: WSGIServer/0.2 CPython/3.6.1
    content-type: application/json
    vary: Accept
    allow: GET, HEAD, OPTIONS
    etag: "8f271d5c5e9add11d3e8e78c761258002e84d65a"
    x-frame-options: SAMEORIGIN
    content-length: 871

body: 
[
  {
    "id": "4835c6de-91ee-471c-9cb4-99941f04f6b8",
    "name": " DOCS",
    "created_at": "2017-11-10T11:36:45.073000Z",
    "updated_at": "2017-11-10T11:36:45.073000Z",
    "additionalField": "test"
  },
  {
    "id": "7fd813e7-7cac-43f5-9534-c0d8476acd57",
    "name": "TMAC",
    "created_at": "2017-11-10T11:36:45.823000Z",
    "updated_at": "2017-11-10T11:36:45.823000Z",
    "additionalField": "test"
  },
  {
    "id": "d5623edc-a202-456b-b6b1-e9b6d0fa3664",
    "name": " TEAM",
    "created_at": "2017-11-10T11:36:45.086000Z",
    "updated_at": "2017-11-10T11:36:45.086000Z",
    "additionalField": "test"
  },
  {
    "id": "d7d231b9-81fc-4847-920a-9881c44f1809",
    "name": " Field Management",
    "created_at": "2017-11-10T11:36:45.249000Z",
    "updated_at": "2017-11-10T11:36:45.249000Z",
    "additionalField": "test"
  },
  {
    "id": "e31b300f-6793-4d4e-b62a-074a906015e2",
    "name": "DMAC",
    "created_at": "2017-11-10T11:36:45.487000Z",
    "updated_at": "2017-11-10T11:36:45.487000Z",
    "additionalField": "test"
  }
]

debug: Could not stringify: 
verbose: Running 'afterAll' hooks
verbose: Wrapping up testing.
complete: 1 passing, 0 failing, 0 errors, 0 skipped, 1 total
complete: Tests took 97ms
verbose: Dredd instance run finished.
verbose: Exiting Dredd process with status '0'.
debug: Using native process.exit() method to terminate the Dredd process.
debug: The backend server process has already terminated

Can you send us failing test in a Pull Request?

In my case, the problem is that the test doesn't fail :)

honzajavorek commented 6 years ago

Hi, this is described in the docs as the Making Dredd Validation Stricter section. Would the suggestions in the section solve this for you?

ido567 commented 6 years ago

Didn't test it yet, but it looks like what I'm trying to achieve. Is there a way to configure it globally? Or by a hook?

honzajavorek commented 6 years ago

It is a sort of limitation/feature of the API description format. The transaction object you get in the hooks has transaction.expected.bodySchema, where you can find the JSON Schema. In a beforeAll or beforeEach hook, you could go and make the schema stricter by appending "additionalProperties": false (ref) wherever you like (recursively?).

ido567 commented 6 years ago

Sound like recursion is the only option. Do you have an example of a recursive function that adds some properties to every expected schema?

honzajavorek commented 6 years ago

I don't know about anything from the top of my head, but try to check out the examples in the sanitation docs.

ido567 commented 6 years ago

OK, I'll check it out. Thank you so much for the help!!

honzajavorek commented 6 years ago

You're welcome! Thank you for using Dredd!