search

vifreefly / kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
MIT License
1.01k stars 155 forks source link

feat: allow preset cookies url #64

Open n-studio opened 3 years ago

n-studio commented 3 years ago

From https://www.selenium.dev/documentation/support_packages/working_with_cookies/

Add Cookie

It is used to add a cookie to the current browsing context. Add Cookie only accepts a set of defined serializable JSON object. Here is the link to the list of accepted JSON key values

First of all, you need to be on the domain that the cookie will be valid for. If you are trying to preset cookies before you start interacting with a site and your homepage is large / takes a while to load an alternative is to find a smaller page on the site (typically the 404 page is small, e.g. http://example.com/some404page)

Selenium recommends to load a small page to load the cookies before interacting with a site.

This PR allows to set a custom url to set cookies.

@start_urls = [{ url: "http://example.com", data: { preset_cookies_url: "http://example.com/some404page" } }]

(Kind of) fixes https://github.com/vifreefly/kimuraframework/issues/63