Selenium error when scraping for too long

[IanQS]

selenium.common.exceptions.TimeoutException: Message: timeout                                         │                                                                                                      
  (Session info: headless chrome=68.0.3440.75)                                                        │                                                                                                      
  (Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.1│                                                                                                      
5.0-30-generic x86_64)                                                                                │                      

[bryantpq]

What This exception happens when the page takes too long to completely load so the content is never returned back for BeautifulSoup to parse.

Plan I will add an async request that will trigger when the content we are looking for the on the page has loaded and signal the driver to stop loading the page. This will reduce some overhead so that we dont have to wait for the page to completely load every time and also will stop this error from happening. After the fix:

• All sites will use Selenium regardless. Its not worth the manpower to check it for every site
• Xpath will be added as it is native to Selenium and BeautifulSoup will be removed because we do not need it anymore
• site_configs.py and associated docs will have to be changed to now support xpath, I will add some docs for how xpath works as well.

[bryantpq]

Closed by #40