search

UrbanInstitute / ipeds-scraper

Download IPEDS complete data files
MIT License
37 stars 8 forks source link

scraper.py throwing error from unfound element #2

Open wbuchanan opened 6 years ago

wbuchanan commented 6 years ago 
Traceback (most recent call last):
  File "scripts/scraper.py", line 47, in <module>
    select = Select(driver.find_element_by_id('contentPlaceHolder_ddlYears'))
  File "/Users/billy/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 344, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/Users/billy/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 858, in find_element
    'value': value})['value']
  File "/Users/billy/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 311, in execute
    self.error_handler.check_response(response)
  File "/Users/billy/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="contentPlaceHolder_ddlYears"]
wbuchanan commented 6 years ago

On a slightly unrelated note, I'm working on developing a Mata-based solution for IPEDS. Figured it might be of interest to some of your team since I know some of them regularly use Stata.

hrecht commented 6 years ago

That error message looks like the html element id="contentPlaceHolder_ddlYears" may not be on the page anymore or there's some issue with selenium finding it. The code is heavily dependent on the page elements staying in place (since there's no permalink to the complete data files, the script has to click the buttons)...try seeing if that element still exists. If the page structure has changed it might require a minor code edit.