search

vifreefly / kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
MIT License
1.01k stars 154 forks source link

Better support for testing #33

Open doutatsu opened 4 years ago

doutatsu commented 4 years ago

As I was writing tests for the scraper I've made, I've realised its not super straightforward at the moment. It would be great to improve on that front:

  1. Add testing section to the documentation, showcasing how to set it up and test in Rails for example
  2. Expand global configuration options. I would have liked to be able to disable delay globally in the test environment, instead of doing this in every scraper I write: @config = { before_request: { delay: 1..2 } } unless Rails.env.test?
  3. Add automatic detection of the test environment. Currently I have to manually set it in the rails_helper: ENV['KIMURAI_ENV'] ||= 'test'
rubyFeedback commented 4 years ago

I think it may be easier to split this into separate suggestions. 3) sounds fairly easy I think, though.