taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 579 forks source link

Error user_info #238

Open aminalgeria opened 4 years ago

aminalgeria commented 4 years ago

I'm trying to scrape a hashtag using the following command twitterscraper #Algerie_Libre_Democratique --csv --profiles However, I get an error from some usernames (the user names are empty (0 followers, 0 likes), they have deleted the tweets but the tweets show up in the search, for example : https://twitter.com/crowded/

INFO: Using proxy 109.203.182.191:80 ERROR: An unknown error occurred! Returning user information gathered so far... Traceback (most recent call last): File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\twitterscraper\query.py", line 322, in query_user_info user_info = query_user_page(INIT_URL_USER.format(u=user)) File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\twitterscraper\query.py", line 291, in query_user_page user_info = User.from_html(html) File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\twitterscraper\user.py", line 100, in from_html return self.from_soup(user_profile_header, user_profile_canopy) File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\twitterscraper\user.py", line 56, in from_soup tweets = tag_prof_nav.find('span', {'class':"ProfileNav-value"})['data-count'] File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\bs4\element.py", line 992, in __getitem__ return self.attrs[key] KeyError: 'data-count' INFO: Got user information from username Crowded Traceback (most recent call last): File "C:\Users\BG\AppData\Local\Programs\Python\Python37-32\Scripts\twitterscraper-script.py", line 11, in <module> load_entry_point('twitterscraper==1.4.0', 'console_scripts', 'twitterscraper')() File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\twitterscraper\main.py", line 139, in main list_users_info = [query_user_info(elem) for elem in list_users] File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\twitterscraper\main.py", line 139, in <listcomp> list_users_info = [query_user_info(elem) for elem in list_users] File "c:\users\bg\appdata\local\programs\python\python37-32\lib\site-packages\twitterscraper\query.py", line 332, in query_user_info return user_info UnboundLocalError: local variable 'user_info' referenced before assignment

cgebe commented 4 years ago

also encountered this issue!

jemorriso commented 4 years ago

And me.

SurajMeena commented 4 years ago

Hi, Are you sure this is the right format to extract hashtag tweets from twitterscrapper ? As far as it's given in readme it should be like twitterscraper "Algerie_Libre_Democratique" --csv --profiles

VolodyaCO commented 4 years ago

I also encountered this issue

djbautista commented 4 years ago

x2

lapp0 commented 4 years ago

Hi all, thanks for your feedback. I've pushed a change that fixes this to https://github.com/taspinar/twitterscraper/pull/302 Please test and let me know if you experience any issues.