npo_watchlist special characters in tvshows problem

solipsist01 commented 6 years ago

Expected behaviour:

That the npo_watchlist plugin can download tvshows from NPO with special characters in their show name and/or episode title.

Actual behaviour:

The script crashes. But only when the tvshow has a special character in it. all the other shows work perfectly fine

Steps to reproduce:

Add https://www.npo.nl/creme-de-la-crooswijk/POMS_S_POWN_12998400 to your watchlist on the NPO website Run flexget execute

Further explanation

The show i have added to the NPO watchlist is called Crème de la Crooswijk The special è in the episode name causes the problem.

In the past i had this exact issue with the following episode of a show calld Typisch https://npo.nl/typisch-wielenpolle/12-03-2018/BV_101386660 The episode name is like "Typisch Wielenpôlle" which has a special character ô

Config:

tasks:
  npo_task:
    npo_watchlist:
      email: redacted@redacted.com
      password: redacted
      remove_accepted: yes
      max_episode_age_days: 30
    accept_all: yes
    exec:
      fail_entries: yes
      auto_escape: yes
      on_output:
        for_accepted:
          - download-npo -o "/output/{{series_name_plain}}" -f "{titel} - {datum} - {aflevering_titel}" {{url}}

Log:

Full debuglogs here: https://pastebin.com/sQKQQDeR The logging below is the console output of the command flexget -L debug execute with above config file. I have a NPO test account with only this show enabled in the watchlist to keep the logs clean.

2018-04-05 04:29 INFO     search_npo    npo_task        Retrieving npo.nl watchlist for redacted@redacted.com
2018-04-05 04:29 VERBOSE  utils.requests npo_task        Waiting 4.80 seconds until next request to npo.nl
2018-04-05 04:29 WARNING  search_npo    npo_task        If max_episode_age_days is 3 days or more, the plugin will still check all series
2018-04-05 04:29 INFO     search_npo    npo_task        Retrieving new episodes for Crème de la Crooswijk
2018-04-05 04:29 VERBOSE  utils.requests npo_task        Waiting 4.77 seconds until next request to npo.nl
2018-04-05 04:29 VERBOSE  details       npo_task        Produced 1 entries.
2018-04-05 04:29 VERBOSE  task          npo_task        ACCEPTED: `Crème de la Crooswijk (POW_03845023)` by accept_all plugin
2018-04-05 04:29 VERBOSE  details       npo_task        Summary - Accepted: 1 (Rejected: 0 Undecided: 0 Failed: 0)
2018-04-05 04:29 VERBOSE  exec          npo_task        Executing: download-npo -o "/output/Creme de la Crooswijk" -f "{titel} - {datum} - {aflevering_titel}" https://www.npo.nl/creme-de-la-crooswijk/02-04-2018/POW_03845023
2018-04-05 04:29 INFO     exec          npo_task        Stdout: Traceback (most recent call last):
  File "/usr/local/bin/download-npo", line 6, in <module>
    download_npo.cli.main()
  File "/usr/local/lib/python2.7/site-packages/download_npo/cli.py", line 330, in main
    options['metaonly'], options['getsubs'], options['quality'])
  File "/usr/local/lib/python2.7/site-packages/download_npo/cli.py", line 204, in download
    error(exc)
  File "/usr/local/lib/python2.7/site-packages/download_npo/cli.py", line 86, in error
    sys.stderr.write(u'error: {}\n'.format(msg))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 48: ordinal not in range(128)
2018-04-05 04:29 ERROR    entry         npo_task        Failed Crème de la Crooswijk (POW_03845023) (exec return code was non-zero)
2018-04-05 04:29 INFO     task          npo_task        Plugin retry_failed has requested task to be ran again after execution has completed.
2018-04-05 04:29 VERBOSE  task          npo_task        FAILED: `Crème de la Crooswijk (POW_03845023)` by exec plugin because exec return code was non-zero
2018-04-05 04:29 INFO     task          npo_task        Rerunning the task in case better resolution can be achieved.
2018-04-05 04:29 INFO     search_npo    npo_task        Retrieving npo.nl watchlist for redacted@redacted.com
2018-04-05 04:29 WARNING  search_npo    npo_task        If max_episode_age_days is 3 days or more, the plugin will still check all series
2018-04-05 04:29 INFO     search_npo    npo_task        Retrieving new episodes for Crème de la Crooswijk
2018-04-05 04:29 VERBOSE  utils.requests npo_task        Waiting 4.65 seconds until next request to npo.nl
2018-04-05 04:29 VERBOSE  details       npo_task        Produced 1 entries.
2018-04-05 04:29 VERBOSE  task          npo_task        REJECTED: `Crème de la Crooswijk (POW_03845023)` by retry_failed plugin because waiting before retrying entry which has failed in the past. (failure reason: exec return code was non-zero)
2018-04-05 04:29 VERBOSE  details       npo_task        Summary - Accepted: 0 (Rejected: 1 Undecided: 0 Failed: 0)

Additional information:

In the logging we can see the actual execute string.

download-npo -o "/output/Creme de la Crooswijk" -f "{titel} - {datum} - {aflevering_titel}" https://www.npo.nl/creme-de-la-crooswijk/02-04-2018/POW_03845023

When i copy this string manually into the commandline, it works without any problem. In my opinion this means it's not an issue with the download-npo downloader.

FlexGet version: 2.13.6
Python version: 2.7.14
Installation method:PIP (inside a docker container)

docker run --name=flexget -d -e TZ=Europe/Amsterdam -v <flexget config path>:/flexget -v <input path>:/input -v <output path>:/output solipsist01/flexget-dockerfile-downloadnpo flexget -L debug execute

Using daemon (yes/no):Tested with and without. Same behaviour
OS and version: Debian 8.10

gazpachoking commented 6 years ago

Looks like it's the download-npo program that's crashing rather than flexget. If the command is running fine manually, it may be something to do with the locale being declared differently for an interactive shell vs however flexget is being run.

solipsist01 commented 6 years ago

You are absolutely right, thank you for pointing me in the right direction :) I have switched from download-npo to Youtube-DL Youtube-DL doesn't have this problem :)

Perhaps it's a good idea to mention or change to this downloader as well on the npo_watchlist documentation located at https://flexget.com/Plugins/npo_watchlist

Youtube-DL can be told to download the best quality as well. Turned out i was downloading mediocre quality ;)

This is my working config

tasks:
  npo_task:
    npo_watchlist:
      email: your@npo_account_mailaddress.com
      password: Y0UrUb3rStr0ngP4ssw0rd
      remove_accepted: yes
      max_episode_age_days: 30
    accept_all: yes
    exec:
      fail_entries: yes
      auto_escape: yes
      on_output:
        for_accepted:
          - youtube-dl -o "/output/{{series_name_plain}}/{{series_name_plain}} - {{series_date | formatdate("%Y-%m-%d") }} - %(title)s.%(ext)s" -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4] {{url}}

This ticket is solved and can be closed :) Thank you.

Flexget / Flexget