search

DKarap / web-driver

crawler that use the webdriver, ghostdriver/phantomJS
0 stars 1 forks source link

GetPageLinks() performance improvement; #7

Closed DKarap closed 10 years ago

DKarap commented 10 years ago
  1. getImgChildElementTextAtributesValue() slows the getLinks process
DKarap commented 10 years ago

Using explicit wait (see commented code in getImgChildElementTextAtributesValue()) doesnt solve the problem due to the webdriver doesnt allow to override the initial set implicity wait.

Solution: set implicit wait to zero or to a very small number

DKarap commented 10 years ago

Solution: 1. set implicit wait to zero or to a very small number

  1. increase pageLoadTime out to make sure the page load correctly
  2. increase THREAD_SLEEP_AFTER_STATE_CHANGE at webdriver