coleifer / micawber

a small library for extracting rich content from urls
http://micawber.readthedocs.org/
MIT License
632 stars 91 forks source link

Custom fetcher #51

Closed spinus closed 9 years ago

spinus commented 9 years ago

Do you plan to add a way to use custom data retrieval method by any chance?

It could be nice because than different methods could be used, like requests library or asyncio in python3.4.

oauthlib (https://oauthlib.readthedocs.org/en/latest/oauth1/client.html) has this implementation and works pretty nice. Example:

client = oauthlib.oauth1.Client('client_key', client_secret='your_secret')
uri, headers, body = client.sign('http://example.com/request_token')
# Here you do a request
# and next you can grab data from response

So in this case it could be

provider = micawber.bootstrap_basic()
url, headers, body = provider.prepare_request(URL)
# Do a request
provider.parse_response(resp)

Or, could be easier maybe to allow override fetch method by using a callback?

provider.request(URL, fetch_callback=my_callback(url, headers, body))

I know I can monkeypatch fetch method but that don't seem to be a good way in a long run.

coleifer commented 9 years ago

Just subclass Provider and write your own fetch method. I suppose if you wanted we could add a parameter to the bootstrap_ methods that accepts a Provider implementation, but I'd rather wait til there's a demonstrable need for such a feature.

spinus commented 9 years ago

I tried to avoid rewriting bootstrap_ methods (recreating providers), but that's fair enough. Thanks.