search

spaam / svtplay-dl

Small command-line program to download videos from some streaming sites.
https://svtplay-dl.se
MIT License
724 stars 122 forks source link

Unicode error when running svtplay-dl from cronjob (Ubuntu 14.04 LTS) #430

Open qnorsten opened 8 years ago

qnorsten commented 8 years ago

I am running svtplay-dl in a cronjob on a server running Ubuntu 14.04 lts. It works most of the time but for some subtitles I get a unicode error. svtplay-dl is running under python3 The command I am running svtplay-dl -S -A --raw-subtitles --remux --silent-semi http://www.svtplay.se/sista-chansen but the video with the problem is http://www.svtplay.se/video/9809554/barnlakarna/barnlakarna-sasong-3-avsnitt-3 So the following command can reproduce the error until the video dissapers. svtplay-dl -S -A --raw-subtitles --remux --silent-semi http://www.svtplay.se/video/9809554/barnlakarna/barnlakarna-sasong-3-avsnitt-3

It seems to be related to the locale of the cronjob as can be seen during the following links http://stackoverflow.com/questions/11735363/python3-unicodeencodeerror-only-when-run-from-crontab http://www.logikdev.com/2010/02/02/locale-settings-for-your-cron-job/

I managed to solve it by adding a LANG=en_US.UTF-8 before the cronjob as the cronjob runs with a different locale in an other case. I do not know if it is fixed in later versions of ubuntu (hopefully it is) or if svtplay-dl can do anything about it. (if it is fixed and there is nothing svtplay-dl can do) feel free to close the issue. At least other people running into the same problem can google it and get a solution.

Locale from terminal:

LANG=en_GB.UTF-8 LANGUAGE= LC_CTYPE="en_GB.UTF-8" LC_NUMERIC="en_GB.UTF-8" LC_TIME="en_GB.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_MONETARY="en_GB.UTF-8" LC_MESSAGES="en_GB.UTF-8" LC_PAPER="en_GB.UTF-8" LC_NAME="en_GB.UTF-8" LC_ADDRESS="en_GB.UTF-8" LC_TELEPHONE="en_GB.UTF-8" LC_MEASUREMENT="en_GB.UTF-8" LC_IDENTIFICATION="en_GB.UTF-8" LC_ALL=

Locale from cronjob (without fix)

LANG= LANGUAGE= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL=

Here is the tracelog:

21/08/2016 18:58:57 Traceback (most recent call last): 21/08/2016 18:58:57 File "/usr/local/bin/svtplay-dl", line 5, in 21/08/2016 18:58:57 pkg_resources.run_script('svtplay-dl==1.2', 'svtplay-dl') 21/08/2016 18:58:57 File "/usr/lib/python3/dist-packages/pkg_resources.py", line 528, in run_script 21/08/2016 18:58:57 self.require(requires)[0].run_script(script_name, ns) 21/08/2016 18:58:57 File "/usr/lib/python3/dist-packages/pkg_resources.py", line 1394, in run_script 21/08/2016 18:58:57 execfile(script_filename, namespace, namespace) 21/08/2016 18:58:57 File "/usr/lib/python3/dist-packages/pkg_resources.py", line 55, in execfile 21/08/2016 18:58:57 exec(compile(open(fn).read(), fn, 'exec'), globs, locs) 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/EGG-INFO/scripts/svtplay-dl", line 7, in 21/08/2016 18:58:57 svtplay_dl.main() 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/svtplay_dl/init.py", line 438, in main 21/08/2016 18:58:57 get_media(url, options) 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/svtplay_dl/init.py", line 169, in get_media 21/08/2016 18:58:57 get_all_episodes(stream, options, url) 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/svtplay_dl/init.py", line 197, in get_all_episodes 21/08/2016 18:58:57 get_one_media(substream, copy.copy(options)) 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/svtplay_dl/init.py", line 270, in get_one_media 21/08/2016 18:58:57 options_subs_dl(subfixes) 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/svtplay_dl/init.py", line 265, in options_subs_dl 21/08/2016 18:58:57 subs[0].download() 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/svtplay_dl/subtitle/init.py", line 45, in download 21/08/2016 18:58:57 self.save_file(subdata, self.subtype) 21/08/2016 18:58:57 File "/usr/local/lib/python3.4/dist-packages/svtplay_dl-1.2-py3.4.egg/svtplay_dl/subtitle/init.py", line 56, in save_file 21/08/2016 18:58:57 file_d.write(data) 21/08/2016 18:58:57 UnicodeEncodeError: 'ascii' codec can't encode character 'xe4' in position 54: ordinal not in range(128)

spaam commented 8 years ago

this happen on a Ubuntu 14.04 system..

qnorsten commented 8 years ago

I updated the problem description know (accidently posted to fast). As I said I managed to solve it, and if there is nothing svtplay-dl can do about it, or if it is solved for newer versions of ubuntu feel free to close it and keep it around as a solution if anyone else is experiencing the same problem.

Edit: also added the locale information know in case something can be done to detect if it runs inside a cronjob.