switch to headless chrome (with selenium inside spider) as scraper alternative for reactive websites

[JulienParis]

roadmap :

• [x] add choice 'parse_reactive' in spider config : front and spider's datamodel
• [x] add selenium + chromedriver on server / local
• [x] write switchers in master_spider
  • [x] switch for 'next_page'
  • [x] switch for 'follow_link'
• [x] redirect data to items
• [x] test locally with reactive and non-reactive websites
• [x] add to prod
  • [x] install chromedriver on server
  • [x] update dependencies on server
  • [x] make some test in prod mode
• [x] add info install chromedriver dans doc install
• [x] merge selenium branch with master

[JulienParis]

references to check :

• scrapy-selenium (need python3) : https://github.com/clemfromspace/scrapy-selenium
• selenium-Python API : https://selenium-python.readthedocs.io/getting-started.html
• selenium in Scrapy : https://stackoverflow.com/questions/17975471/selenium-with-scrapy-for-dynamic-page

[JulienParis]

exemple de site problématique :

• https://www.unccas.org/-banque-d-experiences- : bouton 'next page' n'est pas activé avec .click()de Selenium ( cf : https://github.com/entrepreneur-interet-general/OpenScraper/blob/selenium/openscraper/scraper/masterspider.py , ligne 562 )