coleifer / micawber

a small library for extracting rich content from urls
http://micawber.readthedocs.org/
MIT License
632 stars 91 forks source link

suggestion: requests and responses #75

Closed jvanasco closed 7 years ago

jvanasco commented 7 years ago

i know that requests is a bit of a resource hit (and it's been brought up before), but I wanted to suggest using it as the Provider (or an ancillary option) because it could improve testing.

The responses library (https://github.com/getsentry/responses) lets you easily intercept calls to the requests library to quickly write integrated tests. for example:

expected_payload = {'author_name': 
                    }
as_json = json.dumps(expected_payload)
with responses.RequestsMock() as rsps:
    rsps.add(responses.GET,
             "http://www.youtube.com/oembed",
             body=as_json,
             status=200,
             content_type='text/html',
             )
    result = providers.request('http://www.youtube.com/watch?v=54XHDUOHuzU')
    for (k, v) in expected_payload.items():
        assert result[k] == v

This was a big benefit to us for testing and simulations (and incredibly easy to implement via subclassing), so I wanted to suggest it upstream.

coleifer commented 7 years ago

Ehh, nah, fuck requests. It's definitely possible to subclass the relevant classes in Micawber if you'd like to create a 3rd-party library that uses requests for the backend, however.

jvanasco commented 7 years ago

It's definitely possible to subclass the relevant classes in Micawber if you'd like to create a 3rd-party library that uses requests for the backend, however.

Yeah, that's what we did (just for the testing). We already had requests loaded in the app, so there wasn't a hit to the process size.