jamesturk / scrapelib

⛏ a library for scraping unreliable pages
https://jamesturk.github.io/scrapelib/
BSD 2-Clause "Simplified" License
208 stars 40 forks source link

add `data` to the argument signature of the key_for_request signature #235

Closed fgregg closed 1 year ago

fgregg commented 1 year ago

For some weird websites, we want to be able to cache post requests. Changing this signature allows us to override key_for_request in our local Scraper class to do so.

jamesturk commented 1 year ago

SGTM, looks like it broke a test, glad to merge once ready

fgregg commented 1 year ago

i think it's httpbin being overloaded. when i run the test suite locally, i'll get some errors in tests that interact with httpbin but i can't reproduce when just running that one test.

fgregg commented 1 year ago

https://pypi.org/project/pytest-httpbin/ looks like it might be a smart idea

fgregg commented 1 year ago

@jamesturk, i made the change to use a local httpbin instance and now all tests are passing.

jamesturk commented 1 year ago

Ah thanks for this, looks great!