Context manager and recording/playback.

[bryanhelmig]

Hey guys, love love love this library (we Zapier folk do a lot of API stuff).

What I'd like to see is a pattern like this (I haven't had a chance to dive into the code much yet, so please forgive odd naming and such):

# serializes any request made within the context into a JSON format
with HTTPretty.record('/path/to/hello-world.json'):
    response = requests.post('http://httpbin.org/post', {'hello': 'world!'}) # live!

Now, located in /path/to/hello-world.json would be written a JSON file like the one below.

{
    "request": {
        "method": "POST",
        "uri": "http://httpbin.org/post"
    },
    "response": {
        "status": 200,
        "headers": {
            "Content-Type": "application/json"
        },
        "body": "{\n  \"origin\": \"123.123.123.123\",\n  \"files\": {},\n  \"form\": {\n    \"hello\": \"world!\"\n  },\n  \"url\": \"http://httpbin.org/post\",\n  \"args\": {},\n  \"headers\": {\n    \"Content-Length\": \"14\",\n    \"Accept-Encoding\": \"gzip, deflate, compress\",\n    \"Connection\": \"keep-alive\",\n    \"Accept\": \"*/*\",\n    \"User-Agent\": \"python-requests/0.14.2 CPython/2.7.1 Darwin/11.4.0\",\n    \"Host\": \"httpbin.org\",\n    \"Content-Type\": \"application/x-www-form-urlencoded\"\n  },\n  \"json\": null,\n  \"data\": \"\"\n}"
    }
}

I didn't bother to show full request/response serialization as it would probably be recorded, but I assume it could be partially defined with sensible defaults if you feel like hand rolling JSON and saving a few (a lot) of characters.

# unserialize requests from JSON format and mock requests
with HTTPretty.playback('/path/to/hello-world.json'):
    response = requests.post('http://httpbin.org/post', {'hello': 'world!'}) # fake!

    assert 200 is response.status_code
    assert 'application/json' is response.headers['Content-Type']
    assert 'origin' in response.json
    assert 'form' in response.json

As long as you guys are cool with such a pattern and have no particular pointers on implementation, I'll dig in and make this happen. Its gonna be incredibly useful!

[bryanhelmig]

Other serialization formats like YAML, XML (yuck) and maybe CURL? CURL could be extra useful as you could define hello-world.curl files which would look more like this:

curl -i -X POST -d hello=world! http://httpbin.org/post

HTTP/1.1 200 OK
Content-Type: application/json
Date: Sat, 10 Nov 2012 02:02:20 GMT
Server: gunicorn/0.13.4
Content-Length: 453
Connection: keep-alive

{
  "origin": "123.123.123.123",
  "files": {},
  "form": {
    "hello": "world!"
  },
  "url": "http://httpbin.org/post",
  "args": {},
  "headers": {
    "Content-Length": "12",
    "Connection": "keep-alive",
    "Accept": "*/*",
    "User-Agent": "curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5",
    "Host": "httpbin.org",
    "Content-Type": "application/x-www-form-urlencoded"
  },
  "json": null,
  "data": ""
}

Just a thought!

[bryanhelmig]

First stab at playback, which works: https://github.com/zapier/HTTPretty/commit/6b7b3a1d05373c91bc750f9e7ac58ada93efa1c7

[gabrielfalcao]

@bryanhelmig this idea is GENIUS! I'm implementing it right now

[bryanhelmig]

If you look at vcrpy, a similar library, they don't have two context managers (record() and playback()) just a single cassette() which will record if the file doesn't exist and playback if it does. That's pretty clever as well!

How can I help you @gabrielfalcao?

[gabrielfalcao]

What about:

recording:

with HTTPretty.record('~/work/project/tests/httpbin.json') as request_registry:
    response = requests.post('http://httpbin.org/post', {'hello': 'world!'})
    if response.status_code is 200:
        request_registry.save_chapter(response)

replaying

with HTTPretty.playback('/path/to/hello-world.json') as context:
    response = requests.post('http://httpbin.org/post', {'hello': 'world!'})

    assert 200 is response.status_code
    assert 'application/json' is response.headers['Content-Type']
    assert 'origin' in response.json
    assert 'form' in response.json

or

from httpretty import httprettified

@httprettified.by('/path/to/hello-world.json'):
def test_httpbin_api_integration(context)
    response = requests.post('http://httpbin.org/post', {'hello': 'world!'})

    assert 200 is response.status_code
    assert 'application/json' is response.headers['Content-Type']
    assert 'origin' in response.json
    assert 'form' in response.json

how is that?

[tabo]

Gabriel,

The one with the decorator would be perfect if it does what Bryan suggests:

1. The first time the test is run, it will connect to the site and store a fixture.
2. The second time it will just read the fixture and use the mock.

Some way to force a refresh of all the HTTP fixtures in a test suite, using for example a HTTPRETTY_REFRESH_FIXTURES=1 env var, would be great. This way we could make sure that our fixtures don't rot.

[bryanhelmig]

+1 on the decorator, but be sure to offer it as context manager as well!

[vandersonmota]

You can get some inspiration from here: https://github.com/vcr/vcr

[roycehaynes]

@gabrielfalcao, I can visit this issue if you haven't done so already. I'll be working on a forked copy. Send a pull request when ready.

[gabrielfalcao]

Perfect, I actually never had time to do it

[bjmc]

Anyone know if this issue still under consideration/development?

[roycehaynes]

@bjmc - I think it's dead unless you work on it.

[gabrielfalcao]

gittips are also a great motivation :D

[kevin1024]

Check out vcr.py if you are interested in a project implementing this API :)

[fatuhoku]

+1 for vcr.py implementing the API. I love the concept of replayable fixtures. As GitHub rightly links, the main issues are to overcome values that change from one request to the next such as nonces and signatures, which cause cache misses.