search

everypolitician / scraped

Write declarative scrapers in Ruby
MIT License
8 stars 0 forks source link

add an open-uri-cached strategy #14

Open tmtmtmtm opened 7 years ago

tmtmtmtm commented 7 years ago

For simplicity when developing a scraper, it would be good to have a Strategy that uses open-uri-cached (with an option for where that cache lives) in the simple default manner (i.e. fetches from remote unless the page is already in the cache).

Related Issue

https://github.com/everypolitician/scraped_page/issues/11 — though that is about reading directly from a pre-existing cache.

chrismytton commented 7 years ago

Would also be good to have a simple way to create the archive from the local cache using capture-open-uri-cached (once that's working properly).