search

adrianshort / uk_planning_scraper

A Ruby gem to get planning applications data from UK council websites.
GNU Lesser General Public License v3.0
27 stars 19 forks source link

added a few LAs and use VCR #18

Closed jnicho02 closed 5 years ago

jnicho02 commented 5 years ago

I added a few local authorities and also used VCR during rspec testing. If you haven't previously seen it, it automagically stores the responses from http calls in a yml store so that you can a) have repeatable tests, b) don't repeatedly hammer the source site.

rspec/brighton_spec.rb is still a bit slow, but if you turn off your wifi connection you will see that it gets its info from /vcr_cassettes/for_4_days_of_Brighton_and_Hove.yml

adrianshort commented 5 years ago

Thank you for this. I'll take a closer look at VCR and see how we could improve the scraping tests.

adrianshort commented 5 years ago

Does setting the scrape delay to 0 ever affect the real server or will it always use the VCR cache?

adrianshort commented 5 years ago

There's no need to add unknownsystem tags as they're done automatically.

jnicho02 commented 5 years ago

Does setting the scrape delay to 0 ever affect the real server or will it always use the VCR cache?

VCR should only be mocking http calls, so any other settings/delays should work as normal. VCR will use the cache if it is there, otherwise it will call t'interweb and create a file ready for next time.

adrianshort commented 5 years ago

So when writing test code using this you probably want to leave the delay as the default (10), then use the delay option once you've got the stuff in the cache. But leaving this code as it is (with delay 0) is the right thing to do to speed up the existing tests.