shaikhsajid1111 / twitter-scraper-selenium

Python's package to scrap Twitter's front-end easily
https://pypi.org/project/twitter-scraper-selenium
MIT License
299 stars 46 forks source link

timeout exception #75

Open ihabpalamino opened 1 year ago

ihabpalamino commented 1 year ago

[WDM] - Driver [C:\Users\HP Probook.wdm\drivers\geckodriver\win64\v0.33.0\geckodriver.exe] found in cache 2023-07-13 13:12:40,345 - twitter_scraper_selenium.driver_utils - ERROR - Tweets did not appear!, Try setting headless=False to see what is happening Traceback (most recent call last): File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\twitter_scraper_selenium\driver_utils.py", line 35, in wait_until_tweets_appear WebDriverWait(driver, 80).until(EC.presence_of_element_located( File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 95, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Stacktrace: RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8 WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:183:5 NoSuchElementError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:395:5 element.find/</<@chrome://remote/content/marionette/element.sys.mjs:134:16

shaikhsajid1111 commented 1 year ago

You must be scraping it without authentication. I think it is because of #74

ihabpalamino commented 1 year ago

yes i am doing it without authentification but its a public account here is my code from twitter_scraper_selenium import scrape_profile

scrape_profile(twitter_username="2MInteractive",output_format="csv",browser="firefox",tweets_count=5,filename="microsoft",directory="/home/user/Downloads",headless=False)

and while trying from twitter_scraper_selenium import get_profile_details

twitter_username = "TwitterAPI" filename = "twitter_api_data" get_profile_details(twitter_username=twitter_username, filename=filename) it worked fine

ihabpalamino commented 1 year ago

and got this error [WDM] - Driver [C:\Users\HP Probook.wdm\drivers\geckodriver\win64\v0.33.0\geckodriver.exe] found in cache 2023-07-15 18:37:18,929 - twitter_scraper_selenium.element_finder - ERROR - Error at method find_status : Message: Unable to locate element: a[aria-label][dir] Stacktrace: RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8 WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:187:5 NoSuchElementError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:505:5 element.find/</<@chrome://remote/content/marionette/element.sys.mjs:135:16 Traceback (most recent call last): File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\twitter_scraper_selenium\element_finder.py", line 68, in find_status anchor = tweet.find_element( File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 433, in find_element return self._execute(Command.FIND_CHILD_ELEMENT, {"using": by, "value": value})["value"] File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 410, in _execute return self._parent.execute(command, params) File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 444, in execute self.error_handler.check_response(response) File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 249, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: a[aria-label][dir] Stacktrace: RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8 WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:187:5 NoSuchElementError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:505:5 element.find/</<@chrome://remote/content/marionette/element.sys.mjs:135:16

2023-07-15 18:37:18,938 - ERROR - Error at method fetch_and_store_data : not enough values to unpack (expected 2, got 0) Traceback (most recent call last): File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\twitter_scraper_selenium\profile.py", line 61, in __fetch_and_store_data status, tweet_url = Finder.find_status(tweet) ValueError: not enough values to unpack (expected 2, got 0)