JSONDecodeError: Expecting value: line 1 column 1 (char 0)

RasmusFonseca commented 3 years ago

Running the minimal example crashes with an error:

$ ipython
Python 3.8.3 (default, May 19 2020, 13:54:14) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from twitter_scraper import get_tweets                                                                                        

In [2]: for tweet in get_tweets('twitter', pages=1): 
   ...:     print(tweet['text']) 
   ...:                                                                                                                               
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-2-bd0d769d18f8> in <module>
----> 1 for tweet in get_tweets('twitter', pages=1):
      2     print(tweet['text'])
      3 

~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in get_tweets(query, pages)
    169             pages += -1
    170 
--> 171     yield from gen_tweets(pages)
    172 
    173 

~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in gen_tweets(pages)
     34         while pages > 0:
     35             try:
---> 36                 json_response = request.json()
     37                 html = HTML(
     38                     html=json_response["items_html"], url="bunk", default_encoding="utf-8"

~/opt/miniconda3/lib/python3.8/site-packages/requests/models.py in json(self, **kwargs)
    896                     # used.
    897                     pass
--> 898         return complexjson.loads(self.text, **kwargs)
    899 
    900     @property

~/opt/miniconda3/lib/python3.8/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    355             parse_int is None and parse_float is None and
    356             parse_constant is None and object_pairs_hook is None and not kw):
--> 357         return _default_decoder.decode(s)
    358     if cls is None:
    359         cls = JSONDecoder

~/opt/miniconda3/lib/python3.8/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/opt/miniconda3/lib/python3.8/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Avishay116 commented 3 years ago

Same here - python 3.7.4

gonhonoria commented 3 years ago

same issue. python 3.7.6

yimingStar commented 3 years ago

The response might be empty due to the request failed.

gonhonoria commented 3 years ago

ok, Why would the request failed? I have tried with a bunch of account names but still getting the same error. Has there been any change on the script?

gonhonoria commented 3 years ago

While trying parsing from Hashtags, it works. The issue is specific to parsing profile tweets

yimingStar commented 3 years ago

It is because the url request for the user profile is somehow unavailable. See issue #166, the user-agent that this module is using cannot get the proper response now. Maybe Twitter has banned the user-agent, or maybe Twitter has changed all their website into Client Side Rendering causing request-html failed to get the elements that it usually could.

botbotbotbotboot commented 3 years ago

@yimingStar Agree. Looks like twitter has made some major changes.

botbotbotbotboot commented 3 years ago

If anyone interested in a workaround, please check this out @yimingStar @bisguzar

https://github.com/Foo-Manroot/tweet-feed/issues/1#issuecomment-686635709

Dont know if this method is good enough.

dgnsrekt commented 3 years ago

Here is my attempt at a solution based on @Foo-Manroot instructions. https://gist.github.com/dgnsrekt/8cdb0c10e4a8fe6dcef65ee3a66a4e33 I'm having an issue with getting rate limited. Once, I can get past that issue I can make it nice and pretty and submit a pull request.

magicaltoast commented 3 years ago

Same issue

n0madic commented 3 years ago

Found another source of tweets - widgets at https://publish.twitter.com/ Get JSON from https://syndication.twitter.com/timeline/profile?screen_name=<NAME> The information there is incomplete, for example, there are no likes, retweets, etc. But there is no rate limit.

Nicknameinmel commented 3 years ago

Found another source of tweets - widgets at https://publish.twitter.com/ Get JSON from https://syndication.twitter.com/timeline/profile?screen_name=<NAME> The information there is incomplete, for example, there are no likes, retweets, etc. But there is no rate limit.

Could you tell me how to do that! Many thanks!

n0madic commented 3 years ago

Everything is very simple - we take JSON from the link, take from it the “body” field with the timeline and parse it as usual. JSON also has "minPosition" and "maxPosition" headers for pagination. Implemented in my Go project https://github.com/n0madic/twitter-scraper.

victorDaniloCasta commented 3 years ago

I have the same issue on Python 3.8.2. Everyone know other library for scrapper or know If the new API twitter version allows to get tweets from more than seven days ago? Thanks a lot.