search

vifreefly / kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
MIT License
1.01k stars 155 forks source link

Selenium Chrome Heroku #47

Open muhammedtufekyapan opened 4 years ago

muhammedtufekyapan commented 4 years ago

Hi Everyone,

Is there any way to solve Heroku's problem with Selenium Chrome engine with Kimurai?

I need to use

@engine = :selenium_chrome

I use build packs for Chrome on Heroku but I still gots error related to file path of Chrome and couldn't find any way to define it in spider settings.

Error message from Heroku:

Selenium::WebDriver::Error::WebDriverError: not a file: "/usr/local/bin/chromedriver"

muhammedtufekyapan commented 4 years ago

Hi everyone,

I fix this issue but now I stuck with ChromeDriver on Heroku. I got the following error;

Selenium::WebDriver::Error::WebDriverError: unable to connect to chromedriver 127.0.0.1:9515

Is there anybody knows how to fix this issue?

johnphamvan commented 4 years ago

@muhammedtufekyapan You likely need to use your own Docker container with chromedriver and headless chrome installed on it. Heroku's default stacks do not have chrome or chromedriver. See https://devcenter.heroku.com/articles/stack-packages

bubbaspaarx commented 4 years ago

Have a look here too. Might help with the driver locations for Chrome and Chromedriver. They have to be manually set for Heroku to pick them up. Set them as ENV variables and reference them as per this link. https://milk1000cc.hatenablog.com/entry/2019/11/04/114351