search

MatthewChatham / glassdoor-review-scraper

Scrape reviews from Glassdoor
BSD 2-Clause "Simplified" License
180 stars 252 forks source link

NoSuchElementException #48

Open sophiesc opened 3 years ago

sophiesc commented 3 years ago

It worked well until last week, but then it started showing the following nosuchelementexception for firms with more than 10 reviews (more than one page): selenium.common.exceptions.NoSuchElementException:+Message:+no+such+element:+Unable+to+locate+element:+%7B%22method%22:%22css+selector%22,%22selector%22:%22.paginationPaginationStylenext%22%7D&ie=UTF-8&oe=UTF-8

I tried other codes that people mentioned in this forum who asked about the same issue, but still not working. Is anyone else having the same issue? I changed the time sleep too but didn't work. Perhaps Glassdoor changed their code? This error definitely has something to do with the "more pages" and "go to the next page" functions, but I'm not sure how to fix it. I've been working to fix this for days but unfortunately still not resolved. Any help would be greatly appreciated. Thank you!!!

elaynenh commented 3 years ago

I had the same issue and couldn't figure out how to fix it--I tried messing with the class name passed into the get_current_page function to match what I saw in the browser, but it still couldn't recignize it--so I ended up just hardcoding the page number into the get_current_page function and running it once for each page I needed... luckily I only needed 20 reviews, so it didn't take long. I just commented out the while loop in main so it would only run once. If anybody has a better solution, please share!

bartels50642 commented 3 years ago

Hey sophiesc and elaynenh,

They must have changed their HTML code, because I had scraping code that was working perfectly up until about ~10 days ago. If anyone has a fix for this please let me know!

ankysri commented 3 years ago

Its giving me cannot find element for 'username' so the script is not logging in

ChrisAngelelli commented 3 years ago

Hey, if you are still having this issue the "paginationPaginationStylenext" has been replaced with "pageContainer". You can review the source by inspecting the page element on the desired page.