JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.48k stars 710 forks source link

Twitter searches in 'top' order fail #647

Closed JustAnotherArchivist closed 1 year ago

JustAnotherArchivist commented 1 year ago

Since sometime today, 'top' searches fail entirely. The API endpoint returns 404.

JustAnotherArchivist commented 1 year ago

@PCJ-001 File a complete bug report if it isn't working. Make sure you're using the 'top' search order and the latest dev version.

YouShen12 commented 1 year ago

I can get the code by running in google colab but not in my local pc. The error log is as follow

Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=%28from%3ADrZahidHamidi+OR+from%3AMuhyiddinYassin+OR+from%3AIsmailSabri60+OR+from%3Aanwaribrahim+OR+from%3Achedetofficial+OR+from%3AKhairykj+OR+from%3AHishammuddinH2O+OR+from%3AMukhrizMahathir+OR+from%3An_izzah+OR+from%3Atianchua%29+until%3A2023-01-14+since%3A2023-01-13&count=100&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel: non-200 status code
4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=%28from%3ADrZahidHamidi+OR+from%3AMuhyiddinYassin+OR+from%3AIsmailSabri60+OR+from%3Aanwaribrahim+OR+from%3Achedetofficial+OR+from%3AKhairykj+OR+from%3AHishammuddinH2O+OR+from%3AMukhrizMahathir+OR+from%3An_izzah+OR+from%3Atianchua%29+until%3A2023-01-14+since%3A2023-01-13&count=100&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel failed, giving up.
Traceback (most recent call last):
  File "c:\Users\User\Documents\FYP Test\scrapingpoliticiantweet.py", line 83, in <module>
    for tweet in sntwitter.TwitterSearchScraper(query,top=True).get_items():
  File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\snscrape\modules\twitter.py", line 680, in get_items
    for obj in self._iter_api_data('https://api.twitter.com/2/search/adaptive.json', params, paginationParams, cursor = self._cursor):
  File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\snscrape\modules\twitter.py", line 369, in _iter_api_data
    obj = self._get_api_data(endpoint, reqParams)
  File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\snscrape\modules\twitter.py", line 339, in _get_api_data
    r = self._get(endpoint, params = params, headers = self._apiHeaders, responseOkCallback = self._check_api_response)
  File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\snscrape\base.py", line 216, in _get
    return self._request('GET', *args, **kwargs)
  File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\snscrape\base.py", line 212, in _request
    raise ScraperException(msg)
snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=%28from%3ADrZahidHamidi+OR+from%3AMuhyiddinYassin+OR+from%3AIsmailSabri60+OR+from%3Aanwaribrahim+OR+from%3Achedetofficial+OR+from%3AKhairykj+OR+from%3AHishammuddinH2O+OR+from%3AMukhrizMahathir+OR+from%3An_izzah+OR+from%3Atianchua%29+until%3A2023-01-14+since%3A2023-01-13&count=100&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel failed, giving up.

I have tried uninstalling and reinstalling the snscrape package. Hence is there any way to solve this?

JustAnotherArchivist commented 1 year ago

Locking this to prevent further spam. A new release with this fix will probably follow soon, but if you're in a hurry, use the dev version (see README). If it still isn't working then, please file a complete separate bug report.