JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.5k stars 713 forks source link

Snscrape doesn't work for Twitter! #865

Closed damlitos closed 1 year ago

damlitos commented 1 year ago

Describe the bug

Hi, all of a sudden snscrape has stopped working on collecting tweets. The error is below. Any ideas how to solve it? can it be due a change from Twitter?

How to reproduce

I can reproduce this error in PyCharm by importing snscrape and calling tweet-scraper

Expected behaviour

returns error

CRITICAL:snscrape.base:Errors: blocked (403), blocked (403), blocked (403), blocked (403) snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=Beckley+Psytech+Limited&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.

Screenshots and recordings

No response

Operating system

Windows 10

Python version: output of python3 --version

3.7

snscrape version: output of snscrape --version

0.6.2.20230320

Scraper

TwitterSearchScraper

How are you using snscrape?

Module (import snscrape.modules.something in Python code)

Backtrace

CRITICAL:snscrape.base:Errors: blocked (403), blocked (403), blocked (403), blocked (403) snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=Beckley+Psytech+Limited&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.

Log output

No response

Dump of locals

No response

Additional context

No response

erikcas commented 1 year ago

uhm, did you happen to check if this was already noticed? https://github.com/JustAnotherArchivist/snscrape/issues/846

krestiankin commented 1 year ago

Twitter прикрив API для збору історичних данних і взагалі щось з приходом І Маска там творяться чудеса ))

у мене така ж помилка.

Errors: blocked (403), blocked (403), blocked (403), blocked (403)

erikcas commented 1 year ago

Twitter прикрив API для збору історичних данних і взагалі щось з приходом І Маска там творяться чудеса ))

у мене така ж помилка.

Errors: blocked (403), blocked (403), blocked (403), blocked (403)

:sleeping:

duplicate. see #846

frenchdevartist commented 1 year ago

I see that the OneMillionTweetMap is still working. How do they manage to make it still working ? https://onemilliontweetmap.com/

15thfrank commented 1 year ago

Having the same issue. Been like that since the weekend.

chr-1z commented 1 year ago

me too, same error.

manumactang commented 1 year ago

Same issue here since the weekend !

weitzelleila commented 1 year ago

how to solve this problem, or there is no solution