JSONDecodeError

[lg8897203]

Recently, I got this json decode error in api.py. The error as follow:

Traceback (most recent call last):
  File \"/home/bdclab/Twitter/job_type_keywords_keeping.py\", line 65, in Crawling
    max_id=maxid)
  File \"/home/bdclab/Twitter/twitter_search.py\", line 47, in twitter_search
    tweets = make_twitter_request.make_twitter_request(twitter_api.search.tweets, **kwargs)
  File \"/home/bdclab/Twitter/make_twitter_request.py\", line 55, in make_twitter_request
    return twitter_api_func(*args, **kw)
  File \"/home/bdclab/anaconda3/lib/python3.7/site-packages/twitter/api.py\", line 334, in __call__
    return self._handle_response(req, uri, arg_data, _timeout)
  File \"/home/bdclab/anaconda3/lib/python3.7/site-packages/twitter/api.py\", line 358, in _handle_response
    res = json.loads(data.decode('utf8'))
  File \"/home/bdclab/anaconda3/lib/python3.7/json/__init__.py\", line 348, in loads
    return _default_decoder.decode(s)
  File \"/home/bdclab/anaconda3/lib/python3.7/json/decoder.py\", line 337, in decode
    obj, end = s
 elf.raw_decode(s, idx=_w(s, 0).end())
  File \"/home/bdclab/anaconda3/lib/python3.7/json/decoder.py\", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 505451 (char 505450)

I checked the error position and found nothing was wrong. So confused.

[gregdan3]

I'm also experiencing this issue with high frequency. I suspect the difficulty in finding the error comes from the high rate of backslashes and unicode escape characters in the output; there isn't a match between the given column and the actual column with the error in the document you see.

[gregdan3]

Followup: I have not observed this issue in ~4 hours of work on latest master. Looks to have already been resolved; can't prove a negative, but it's definitely more stable

Major props to this project for not altering their API in all the time since the last PyPI update!

[gregdan3]

Followup: this is not fixed. I happen to not run into it for an extended period of time.

Twitter is sending incomplete JSON, sometimes in the middle of a key or object. I can't trigger it with any consistency, but I have an example object of what twitter (library) threw.

doc.txt

[lg8897203]

Followup: this is not fixed. I happen to not run into it for an extended period of time.

Twitter is sending incomplete JSON, sometimes in the middle of a key or object. I can't trigger it with any consistency, but I have an example object of what twitter (library) threw.

doc.txt

Hi @gregdan3

I have already fixed this issue.

The problem is caused by the error called "http_client.IncompleteRead". I used the retry mechanism "_handle_response_with_retry in api.py" to deal with this problem. When the program catch this error, a retry is triggered.

[boogheta]

Hi @lg8897203, interesting that the retry mechanism would fix this problem. Can you precise the way you used it? Did you just call the Twitter class using the retry=True argument, or did you also edit the library's code so that handle_response_with_retries catches more cases of exceptions?

[lg8897203]

Hi @boogheta,

I changed the way of requesting the url from urllib.request.urlooen() to requests. At last, the problem has finally fixed.

[boogheta]

Hello @lg8897203, do you mean you added requests as a dependency within a fork of the lib?