toutv fetch sometimes error out with "Expecting value"

[gboudreau]

$ toutv fetch Show
Error: cannot fetch "Épisode 01": Expecting value: line 1 column 1 (char 0)
Error: cannot fetch "Épisode 08": Expecting value: line 1 column 1 (char 0)
Error: cannot fetch "Épisode 15": Expecting value: line 1 column 1 (char 0)

Repeating the same toutv fetch command will often work as expected. Error seems to be pretty random.

[simark]

Is it because of the É?

[gboudreau]

I doubt. Sometimes, it works without a problem. Other times, it outputs this error. For the exact same command.

[gboudreau]

Today, I got:

Error: cannot fetch "Épisode 19": Expecting value: line 1 column 1 (char 0)
Error: cannot fetch "Épisode 02": Expecting value: line 1 column 1 (char 0)
Error: cannot fetch "Épisode 11": Expecting value: line 1 column 1 (char 0)

[simark]

For which show?

This is part of a cron job, isn't it? We would need some logging, so if you can add -v, it would be useful. We'd get a backtrace (hopefully). If having all this output in a cron job isn't ideal, we can add a "--log-file" option to redirect the logging output to a file..

[simark]

I have added --log-file in this branch: https://github.com/simark/pytoutv/tree/log-file

Example: toutv -v --log-file toutv.log fetch Infoman S17E06

Right now it will append to the file if it already exists.

[simark]

While working on pytoutv, I got this:

Unknown exception: <class 'simplejson.scanner.JSONDecodeError'>: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/home/simark/src/pytoutv/toutvcli/app.py", line 110, in run
    args.func(args)
  File "/home/simark/src/pytoutv/toutvcli/app.py", line 499, in _command_fetch
    self._fetch_episode(episode, output_dir=output_dir, quality=quality, bitrate=bitrate, overwrite=overwrite)
  File "/home/simark/src/pytoutv/toutvcli/app.py", line 715, in _fetch_episode
    qualities = episode.get_available_qualities()
  File "/home/simark/src/pytoutv/toutv/bos.py", line 544, in get_available_qualities
    playlist, cookies = self.get_playlist_cookies()
  File "/home/simark/src/pytoutv/toutv/bos.py", line 532, in get_playlist_cookies
    url = self._get_playlist_url()
  File "/home/simark/src/pytoutv/toutv/bos.py", line 524, in _get_playlist_url
    response_obj = r.json()
  File "/home/simark/src/pytoutv-env/lib/python3.5/site-packages/requests-2.12.1-py3.5.egg/requests/models.py", line 833, in json
    self.content.decode(encoding), **kwargs
  File "/home/simark/src/pytoutv-env/lib/python3.5/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/home/simark/src/pytoutv-env/lib/python3.5/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/home/simark/src/pytoutv-env/lib/python3.5/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

But as you say, it's intermittent. I retried the command and now it works... but at least now we got a backtrace!

[simark]

Since it complains that it expects a value at character 0 (line 1, column 1), it sounds like sometimes we get an empty reply...

If that's indeed the problem, it can be faked with the following change.

diff --git a/toutv/bos.py b/toutv/bos.py
index be2318c..3513551 100644
--- a/toutv/bos.py
+++ b/toutv/bos.py
@@ -521,6 +521,7 @@ class Episode(_Bo, _ThumbnailProvider):
         params['idMedia'] = self.PID

         r = self._do_request(url, params=params)
+        r._content = b''
         response_obj = r.json()

         if response_obj['errorCode']:

[gboudreau]

The code above create a different exception.

Unknown exception: <class 'json.decoder.JSONDecodeError'>: Expecting value: line 1 column 1 (char 0)

vs what I had:

Error: cannot fetch "Épisode 19": Expecting value: line 1 column 1 (char 0)

So yeah, it's probably an issue with a empty response, or something similar, but it's probably not in Episode._get_playlist_url()

[simark]

It's because you are downloading a complete show, so it's a slightly different code path. But the exception is probably the same. See:

$ toutv fetch "en audition avec simon"
Error: cannot fetch "Épisode 01": Expecting value: line 1 column 1 (char 0)
Error: cannot fetch "Épisode 02": Expecting value: line 1 column 1 (char 0)
Error: cannot fetch "Épisode 03": Expecting value: line 1 column 1 (char 0)
Error: cannot fetch "Épisode 04": Expecting value: line 1 column 1 (char 0)

vs

$ toutv fetch "en audition avec simon" S02E36
Unknown exception: <class 'simplejson.scanner.JSONDecodeError'>: Expecting value: line 1 column 1 (char 0)

[gboudreau]

Indeed.

[simark]

So yeah, it's probably an issue with a empty response, or something similar, but it's probably not in Episode._get_playlist_url()

Well, the backtrace I posted earlier doesn't lie, it did happen in _get_playlist_url. However, it's possible that other replies to come back empty too.

[gboudreau]

Yep, upon more testing, realized that. Implement some simple retry for that method; will see if that fixes my issues.

[gboudreau]

Not seeing this error anymore, but will sometimes get a few WARNING:root:GetPlaylistURL failed. Will retry... in my logs, which means this fix worked! Hooray.

[simark]

:+1: