JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.51k stars 712 forks source link

Errors: blocked (403), blocked (403), blocked (403), blocked (403) #982

Closed ghost closed 1 year ago

ghost commented 1 year ago

Describe the bug

Hello, I'm trying to scrape tweets for the last 3 months mentioning a certain keyword. It used to work fine now it doesn't. I'm using Python.

How to reproduce

import snscrape.modules.twitter as sntwitter for index, tweet in enumerate(sntwitter.TwitterSearchScraper(str(fetching_keyword).strip() + ' since:' + starting_date + ' until:' + end_date, maxEmptyPages = 99999999).get_items()): tweets_list.append([tweet.id, tweet.date, tweet.user.username, tweet.user.location, tweet.lang, tweet.content, tweet.replyCount, tweet.retweetCount, tweet.likeCount, tweet.quoteCount, tweet.outlinks, tweet.tcooutlinks, tweet.retweetedTweet, tweet.quotedTweet, tweet.mentionedUsers])

Expected behaviour

Tweets should be scraped as normal but instead, I receive an error.

Screenshots and recordings

No response

Operating system

Windows 11

Python version: output of python3 --version

3.8.16

snscrape version: output of snscrape --version

0.7.0.20230622

Scraper

TwitterSearchScraper

How are you using snscrape?

Module (import snscrape.modules.something in Python code)

Backtrace

No response

Log output

No response

Dump of locals

ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=Bank+of+Canada+since%3A2023-03-20+until%3A2023-6-25&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.

Additional context

No response

JustAnotherArchivist commented 1 year ago

That error is impossible with version 0.7.0. You are using something older.

doffn commented 1 year ago

i have encountered that problem too, i am using snscrape 0.7.0.20230622

JustAnotherArchivist commented 1 year ago

@doffn No, as I said, that exact error is impossible with version 0.7.0. You probably have two (or more) installations of snscrape and are using an older one than you think.