parroty / exvcr

HTTP request/response recording library for elixir, inspired by VCR.
MIT License
720 stars 131 forks source link

Things that should break don't break... #135

Closed hickscorp closed 6 years ago

hickscorp commented 6 years ago

We're evaluating ExVCR and we had the funniest surprise.

After recording a bunch of requests and removing connectivity to the external service, all our tests stayed green. However, we noticed that things that should turn red don't. Namelly:

Is that a known behaviour? How to make sure that the results are reliable?

Example 1:

[
  {
    "request": {
      "body": "",
      "headers": [],
      "method": "get",
      "options": [],
      "request_body": "",
      "url": "http://localhost:5001/api/v0/pin/rm?arg=QmQChpnJJ5am4HRiW7b1KZtBEBeWy3azovVMCL3xsFVUL3"
    },
    "response": {
      "binary": false,
      "body": "{\"Pins\":[\"QmQChpnJJ5am4HRiW7b1KZtBEBeWy3azovVMCL3xsFVUL3\"]}\n",
      "headers": {
        "Access-Control-Allow-Headers": "X-Stream-Output, X-Chunked-Output, X-Content-Length",
        "Access-Control-Expose-Headers": "X-Stream-Output, X-Chunked-Output, X-Content-Length",
        "Content-Type": "application/json",
        "Control-Allow-Methods": "PUT",
        "Server": "go-ipfs/0.4.13",
        "Trailer": "X-Stream-Error",
        "Vary": "Origin",
        "Date": "Tue, 17 Apr 2018 20:22:20 GMT",
        "Transfer-Encoding": "chunked"
      },
      "status_code": 200,
      "type": "ok"
    }
  }
]

If we change the value of url after arg=, the tests stay green.

Example 2:

[
  {
    "request": {
      "body": "{:multipart, [{:file, \"test/fixtures/Very Nice Great Success.jpg\", {\"form-data\", [name: \"Very Nice Great Success.jpg\", filename: \"test/fixtures/Very Nice Great Success.jpg\"]}, []}]}",
      "headers": [],
      "method": "post",
      "options": [],
      "request_body": "",
      "url": "http://localhost:5001/api/v0/add"
    },
    "response": {
      "binary": false,
      "body": "{\"Name\":\"QmQChpnJJ5am4HRiW7b1KZtBEBeWy3azovVMCL3xsFVUL3\",\"Hash\":\"QmQChpnJJ5am4HRiW7b1KZtBEBeWy3azovVMCL3xsFVUL3\",\"Size\":\"44722\"}\n",
      "headers": {
        "Access-Control-Allow-Headers": "X-Stream-Output, X-Chunked-Output, X-Content-Length",
        "Access-Control-Expose-Headers": "X-Stream-Output, X-Chunked-Output, X-Content-Length",
        "Content-Type": "application/json",
        "Control-Allow-Methods": "PUT",
        "Server": "go-ipfs/0.4.13",
        "Trailer": "X-Stream-Error",
        "Vary": "Origin",
        "X-Chunked-Output": "1",
        "Date": "Tue, 17 Apr 2018 20:22:20 GMT",
        "Transfer-Encoding": "chunked"
      },
      "status_code": 200,
      "type": "ok"
    }
  }
]

If in our source code we completelly remove the multipart attachement, everythign stays green.

hickscorp commented 6 years ago

Closing issue - my bad.

Initially I thought that match_requests_on: [:query, :request_body] was a Config. I realised that it had to be used as parameters of the use_cassette macro.