[Optimization, N/A] print response body when test fails

[hornc]

FYI, Here is the output of: rspec spec/data/example_specs.rb which runs the test specs that show now apivore handles various kinds of failures.

API testing scenarios
  unimplemented path
    fails (FAILED - 1)
  mismatched property type
XXXXXXXXXXXXXXXX
Response body for 'get /api/services/1.json'
{
  "id": 1,
  "name": "hello world"
}
XXXXXXXXXXXXXXXX
    fails (FAILED - 2)
  unexpected http response
XXXXXXXXXXXXXXXX
Response body for 'get /api/services.json'
[
  {
    "id": 1,
    "name": "hello world"
  }
]
XXXXXXXXXXXXXXXX
    fails (FAILED - 3)
  extra properties
XXXXXXXXXXXXXXXX
Response body for 'get /api/services.json'
[
  {
    "id": 1,
    "name": "hello world"
  }
]
XXXXXXXXXXXXXXXX
    fails (FAILED - 4)
XXXXXXXXXXXXXXXX
Response body for 'get /api/services/1.json'
{
  "id": 1,
  "name": "hello world"
}
XXXXXXXXXXXXXXXX
    also fails (FAILED - 5)
  missing required
XXXXXXXXXXXXXXXX
Response body for 'get /api/services.json'
[
  {
    "id": 1,
    "name": "hello world"
  }
]
XXXXXXXXXXXXXXXX
    fails (FAILED - 6)
XXXXXXXXXXXXXXXX
Response body for 'get /api/services/1.json'
{
  "id": 1,
  "name": "hello world"
}
XXXXXXXXXXXXXXXX
    also fails (FAILED - 7)
  missing non-required
    passes
    also passes
  fails custom validation
    passes
    fails (FAILED - 8)

Failures:

  1) API testing scenarios unimplemented path fails
     Failure/Error: expect(subject).to validate(:get, "/not_implemented.json", 200)
       Path /not_implemented.json did not respond with expected status code. Expected 200 got 404
       Response body: 
     # ./spec/data/example_specs.rb:10:in `block (3 levels) in <top (required)>'

  2) API testing scenarios mismatched property type fails
     Failure/Error: expect(subject).to validate(:get, "/services/{id}.json", 200, { "id" => 1 })
        '/api/services/1.json#/name' of type String did not match one or more of the following types: integer, null 
     # ./spec/data/example_specs.rb:17:in `block (3 levels) in <top (required)>'

  3) API testing scenarios unexpected http response fails
     Failure/Error: expect(subject).to validate(:get, "/services.json", 222)
       Path /services.json did not respond with expected status code. Expected 222 got 200
       Response body: [{"id":1,"name":"hello world"}]
     # ./spec/data/example_specs.rb:25:in `block (3 levels) in <top (required)>'

  4) API testing scenarios extra properties fails
     Failure/Error: expect(subject).to validate(:get, "/services.json", 200)
        '/api/services.json#/0' contains additional properties ["name"] outside of the schema when none are allowed 
     # ./spec/data/example_specs.rb:33:in `block (3 levels) in <top (required)>'

  5) API testing scenarios extra properties also fails
     Failure/Error: expect(subject).to validate(:get, "/services/{id}.json", 200, { "id" => 1})
        '/api/services/1.json#/' contains additional properties ["name"] outside of the schema when none are allowed 
     # ./spec/data/example_specs.rb:37:in `block (3 levels) in <top (required)>'

  6) API testing scenarios missing required fails
     Failure/Error: expect(subject).to validate(:get, "/services.json", 200)
        '/api/services.json#/0' did not contain a required property of 'test_required' 
     # ./spec/data/example_specs.rb:45:in `block (3 levels) in <top (required)>'

  7) API testing scenarios missing required also fails
     Failure/Error: expect(subject).to validate(:get, "/services/{id}.json", 200, { "id" => 1})
        '/api/services/1.json#/' did not contain a required property of 'test_required' 
     # ./spec/data/example_specs.rb:49:in `block (3 levels) in <top (required)>'

  8) API testing scenarios fails custom validation fails
     Failure/Error: expect(subject).to conform_to(Apivore::CustomSchemaValidator::WF_SCHEMA)
       The property '#/definitions/service' did not contain a required property of 'type' in schema file:///Users/charles/westfield/apivore/data/custom_schemata/westfield_api_standards.json#
     # ./spec/data/example_specs.rb:72:in `block (3 levels) in <top (required)>'

Finished in 0.26386 seconds (files took 1.51 seconds to load)
11 examples, 8 failures

Failed examples:

rspec ./spec/data/example_specs.rb:9 # API testing scenarios unimplemented path fails
rspec ./spec/data/example_specs.rb:16 # API testing scenarios mismatched property type fails
rspec ./spec/data/example_specs.rb:24 # API testing scenarios unexpected http response fails
rspec ./spec/data/example_specs.rb:32 # API testing scenarios extra properties fails
rspec ./spec/data/example_specs.rb:36 # API testing scenarios extra properties also fails
rspec ./spec/data/example_specs.rb:44 # API testing scenarios missing required fails
rspec ./spec/data/example_specs.rb:48 # API testing scenarios missing required also fails
rspec ./spec/data/example_specs.rb:71 # API testing scenarios fails custom validation fails

[adamcohen]

@hornc looks good to me, we get a print out of the response body for each failure, exactly as I wanted

[hornc]

small update to this PR here: https://github.com/westfieldlabs/apivore/pull/74 closing to merge #74 instead.