Scraper with Selenium config behaves differently with or without headless option

entrepreneur-interet-general / OpenScraper

An open source webapp for scraping: towards a public service for webscraping

http://www.cis-openscraper.com/

MIT License

93 stars 22 forks source link

Scraper with Selenium config behaves differently with or without headless option #65

Open retdop opened 5 years ago

retdop commented 5 years ago

Crawling/Testing not working with that config.

screenshot from 2019-02-20 18-18-44

The xpath doesn't seem to find any element. screenshot from 2019-02-20 18-20-31

Crawler is working when I comment options_selenium.add_argument('headless') in masterspider.py line 101.

This is very weird as chromedriver is supposed to behave identically with or without headless.

PagesJaunes is known to have implemented scraping protections . This may be related.

retdop commented 5 years ago

Enhancement suggestion:

add an option to start chrome as headless
in the configuration form of a spider, add the option to write your own header

See https://medium.com/@addnab/puppeteer-quick-fix-for-differences-between-headless-and-headful-versions-of-a-webpage-5b168bd5fe4a

retdop commented 5 years ago

After some research, it seems really complicated to change the header of requests in Selenium (the easiest way is to use a local proxy...).

Also, there seems to be quite a few differences between chrome and chrome headless. It may be on purpose. So the best solution would actually be to propose an option to use Firefox (geckodriver) instead of Chrome, which actually solve the problem here (tested).