codeforsanjose / city-agenda-scraper

9 stars 16 forks source link

Allow for headlesss firefox in Selenium #42

Closed dntrply closed 3 years ago

dntrply commented 3 years ago

Development environments and/or production environments may not have a GUI/display interface (i.e. - all work is done on the terminal). In this case, redesigning the invocation to a headless firefox would allow execution on systems with and without a GUI/display. See How to make Firefox headless programmatically in Selenium with Python? for a similar reported issue.

Clearly - the solution should be such that it works in systems without a GUI as well as systems with a GUI

Example error upon invocation:

Traceback (most recent call last):
  File "/mnt/d/python_dev/volunteer/city-agenda-scraper/Legistar_scraper/Legistar_Selenium.py", line 108, in <module>
    scrape_meetings(current_city)
  File "/mnt/d/python_dev/volunteer/city-agenda-scraper/Legistar_scraper/Legistar_Selenium.py", line 36, in scrape_meetings
    driver = webdriver.Firefox()
  File "/home/user1/anaconda3/envs/cas/lib/python3.9/site-packages/selenium/webdriver/firefox/webdriver.py", line 170, in __init__
    RemoteWebDriver.__init__(
  File "/home/user1/anaconda3/envs/cas/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/user1/anaconda3/envs/cas/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/user1/anaconda3/envs/cas/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/user1/anaconda3/envs/cas/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)

#-------------------------------------------------------------------------------
selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status 1