search

MatthewChatham / glassdoor-review-scraper

Scrape reviews from Glassdoor
BSD 2-Clause "Simplified" License
179 stars 252 forks source link

No Such Element Error #25

Open ghost opened 5 years ago

ghost commented 5 years ago

I'm getting a no such element error that I don't know how to fix. Help please.

2019-09-11 08:48:58,961 INFO 377    :main.py(1824) - Configuring browser

DevTools listening on ws://127.0.0.1:#####/devtools/browser/cccb51a6-3dc2-4f06-90db-660d#########

2019-09-11 08:49:03,942 INFO 419    :main.py(1824) - Scraping up to 1000 reviews.
2019-09-11 08:49:03,946 INFO 358    :main.py(1824) - Signing in to email@gmail.com
2019-09-11 08:49:06,541 INFO 339    :main.py(1824) - Navigating to company reviews
2019-09-11 08:49:12,674 INFO 286    :main.py(1824) - Extracting reviews from page 1
2019-09-11 08:49:12,696 INFO 291    :main.py(1824) - Found 10 reviews on page 1
2019-09-11 08:49:12,840 WARNING 126    :main.py(1824) - Failed to scrape employee_title
Traceback (most recent call last):
  File "main.py", line 461, in <module>
    main()
  File "main.py", line 441, in main
    reviews_df = extract_from_page()
  File "main.py", line 295, in extract_from_page
    data = extract_review(review)
  File "main.py", line 281, in extract_review
    res[field] = scrape(field, review, author)
  File "main.py", line 264, in scrape
    return fdict[field](review)
  File "main.py", line 156, in scrape_years
    'reviewBodyCell').find_element_by_tag_name('p')
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 305, in find_element_by_tag_name
    return self.find_element(by=By.TAG_NAME, value=name)
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 659, in find_element
    {"using": by, "value": value})['value']
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"p"}
  (Session info: headless chrome=77.0.3865.75)
sachinchaturvedi93 commented 5 years ago 
def scrape_years(review):
    try:
        res = review.find_element_by_class_name('mainText').text.strip('"')
    except Exception:
        res = np.nan
    return res

Replace the scrape_years function by this function.