SerpentAI / selenium-respectful

Minimalist Selenium WebDriver wrapper to work within rate limits of any amount of websites simultaneously. Parallel processing friendly.
Other
51 stars 10 forks source link

Cannot use with RespectfulRequester #2

Open janmyszkier opened 5 years ago

janmyszkier commented 5 years ago

Sample code:

class FirstWebDriverTest(unittest.TestCase):

  def setUp(self):

    #initialize Webdriver with Request Limit
    self.driver = RespectfulWebdriver(webdriver=webdriver.Chrome())
    self.driver.register_realm("Shopify", max_requests=2, timespan=1)

    #setup Requests in same reals to figure out error code of the site without exceeding the limit
    self.rr = RespectfulRequester()
    self.rr.register_realm("Shopify", max_requests=2, timespan=1)

  def test_check_statuses(self):
    url = 'https://someshopify.myshopify.com/somepage'
    r = self.rr.get(url, realms=["Shopify"], wait=True, allowRedirects=False)
    print(r.status_code)

    if(r.status_code == 200):
      driver = self.driver
      driver.get(url, realms=["Shopify"], wait=True)

  def tearDown(self):
    self.driver.close()

if __name__ == '__main__':
    unittest.main()

When I do not use requests_respectful the RespectfulWebdriver is working nice, but if I want to force the same limit on both RespectfulWebdriver and RespectfulRequester then the RespectfulWebdriver doesn't even load the page. Browser opens, but it keeps waiting and waiting.

janmyszkier commented 5 years ago

@nbrochu is this something you can look into?