Altimis / Scweet

A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers, user info, images...
MIT License
1.05k stars 224 forks source link

get_user_follow fails since new chrome version #113

Open PairplaneN opened 2 years ago

PairplaneN commented 2 years ago

Hi, When scraping followers and followings, I receive the following error (since the update of chrome on my linux pc from version 98.xxx to 99.xxx):

Traceback (most recent call last):

File "~/Scweet/utils.py", line 385, in get_users_follow log_in(driver, env, wait=wait) File "~/Scweet/utils.py", line 305, in log_in email_el = driver.find_element(By.XPATH, email_xpath) File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 1244, in find_element return self.execute(Command.FIND_ELEMENT, { File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute self.error_handler.check_response(response) File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//input[@autocomplete="username"]"} (Session info: headless chrome=99.0.4844.51) Stacktrace:

0 0x5613b9b24233

1 0x5613b987a658

2 0x5613b98b0941

3 0x5613b98b0b01

4 0x5613b98e3a34

5 0x5613b98ce06d

6 0x5613b98e17bb

7 0x5613b98cdf33

8 0x5613b98a3c8a

9 0x5613b98a4d55

10 0x5613b9b54c5d

11 0x5613b9b6e17e

12 0x5613b9b569c5

13 0x5613b9b6ec95

14 0x5613b9b4a900

15 0x5613b9b89e28

16 0x5613b9b89fa8

17 0x5613b9ba36ad

18 0x7f7219033609

Thank you for looking into this problem.

rosscg commented 2 years ago

I've added Firefox support in the fork here: https://github.com/rosscg/Scweet/

It's set as the default in get_users_follow so it can run without changing anything else.

ShoufaChen commented 2 years ago

Hi, @rosscg

Thanks for your contribution. I found that you import

from .const import get_username, get_password, get_email, get_geckdriver_path

but .const does not contains get_geckdriver_path.

Could you help me?

rosscg commented 2 years ago

@ShoufaChen Yes that's an import I replaced (so shouldn't be there). I've updated the repo so either use the latest on my fork or just remove ", get_geckdriver_path" from that line in utils.py

ShoufaChen commented 2 years ago

Thank you @rosscg.

PairplaneN commented 2 years ago

Thank you @rosscg and @ShoufaChen, it's working again (The firefox version, not yet the chrome version).