Update to Python 3?

[FlyveHest]

I am using Linuxserver.ios docker image for NZBGet, and they recently dropped support for Python2 (completely removed it), and running the VideoSort plugin on Python3 gives the following error

VideoSort: SyntaxError: invalid syntax
VideoSort: ^
VideoSort: ep_prefix = episode_separator if episode_num_all <> '' else ''
VideoSort: File "/workdir/scripts/videosort/VideoSort.py", line 814

[ohyeahsure]

I'm seeing a similar error, just the line number being different (851 in my case)

[shibuyastarbucks]

Also got syntax error on line 851.

[hugbug]

VideoSort requires Python 2. The script description clearly states that. If anyone can port it to Python 3 (and preferably keep it compatible with Python 2), please submit a pull request.

[FlyveHest]

Yes, it does, but Python2 was EOLed in january. (Which you probably knew)

Personally I rolled back the docker container I was using to an older version, and have got the script running again, but it would be really nice to get the script ported to Python3.

I'm not a Python dev, so I don't think I can be of much help, unfortunately .. But just for fun, is there an easy way to run the script standalone with the testdata in the repo?

[hugbug]

Yes, just run testsort.py from terminal.

[shibuyastarbucks]

linuxserver.io provided a workaround to use customized script inside the container: https://blog.linuxserver.io/2019/09/14/customizing-our-containers/

For the time being, a temporary fix is to log into the NZBGet container terminal, and run script below.

mkdir -p /config/custom-cont-init.d
cat <<EOF >/config/custom-cont-init.d/python2.sh
#!/bin/bash
apk add --no-cache python2
EOF
chmod +x /config/custom-cont-init.d/python2.sh

And then restart the container.

[djl]

My fork has Python 3 support. As the most recent commit implies, it's not pretty. I just wanted to get something working for personal use. The tests haven't been updated and it probably doesn't run on Python 2 but if anyone wants to use it as a starting point for a real port/PR then feel free.

[timgilbert]

I started to work on porting this to python 3 too, and managed to get to a reasonable state using shiv to produce a .pyz file out of this, but then I realized that nzbget itself can't read the extension script metadata out of a pyz file. I think my code would be usable with some tweaking though, and I can submit it as a PR once I have more time to work on it. A few of the unit tests no longer pass due to underlying changes in the latest version of the guessit library.

[jouster1974]

Ive also managed to install an older version of BZBGet that gets thigns working agian but guess going forwarded there will need to be an update of some sort to videosort

[taalas]

VideoSort requires Python 2. The script description clearly states that. If anyone can port it to Python 3 (and preferably keep it compatible with Python 2), please submit a pull request.

Just out of curiosity: you have not planned to port the script to a more recent Python version yourself? Is the project abandoned?

[hugbug]

I've made some changes for python 3 compatibility. In my limited tests it worked fine with both python 3 and 2. All built-in tests pass.

Please try this version from branch python3. After a positive feedback I can publish it as a new release.

[timgilbert]

Thanks @hugbug. When I try to just run the python3 branch (from a git checkout) I get these errors. This is via the linuxserver docker image, but I also see the same errors just trying to run VideoSort.py locally on my Mac with python 3.8.5:

info | Thu Aug 06 2020 12:56:37 | VideoSort: AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'
info | Thu Aug 06 2020 12:56:37 | VideoSort:     register_loader_type(importlib_bootstrap.SourceFileLoader, DefaultProvider)
info | Thu Aug 06 2020 12:56:37 | VideoSort:   File "/downloads/scripts/VideoSort/lib/pkg_resources.py", line 1341, in <module>
info | Thu Aug 06 2020 12:56:37 | VideoSort:     from pkg_resources import iter_entry_points, EntryPoint
info | Thu Aug 06 2020 12:56:37 | VideoSort:   File "/downloads/scripts/VideoSort/lib/babelfish/converters/__init__.py", line 6, in <module>
info | Thu Aug 06 2020 12:56:37 | VideoSort:      from .converters import (LanguageConverter,  LanguageReverseConverter, LanguageEquivalenceConverter,  CountryConverter,
info | Thu Aug 06 2020 12:56:37 | VideoSort:   File "/downloads/scripts/VideoSort/lib/babelfish/__init__.py", line 20, in <module>
info | Thu Aug 06 2020 12:56:37 | VideoSort:     import babelfish

I didn't see an obvious way to create a release bundle / zip file from the git repo, is there anything I should be doing besides just checking it out in my ScriptDir directory?

[timgilbert]

Following up, I'm guessing you may want to upgrade the guessit dependency from 2.1.4 to the most recent version, 3.1.1. I did this in my branch and it seemed to work but it broke some unit tests. The root of the problem, though, is probably that all of the script's dependencies are packaged along with it in the git repo, versus using a requirements.txt / pyproject.toml / etc to manage dependencies.

[hugbug]

I've made the changes on an older Mac with Python 3.3 where it worked. I've just tried this with a newer Python and it fails with the same error. I'll look into this.

I'm guessing you may want to upgrade the guessit dependency from 2.1.4 to the most recent version, 3.1.1. I did this in my branch and it seemed to work but it broke some unit tests.

The tests fail because you updated guessit. VideoSort doesn't work properly most likely. It seems something important has changed in guessit and requires changes in VideoSort. No, I don't want to update all dependencies just because there are new versions. Such updates often cause issues like in this case.

[hugbug]

I've made some changes. Please try the newer version. Thanks.

[timgilbert]

@hugbug: thanks for the fast update. I tried the latest commits and everything seems to work great for me (running in the latest linuxserver/nzbget docker image). I also tried just running the script in both python 2.7.16 and python 3.8.5 on my mac and nothing blew up on import, though I didn't do any extensive testing.

One oddity was that when I first ran it I got an error about not being able to find MoviesDir, but after I went to Settings > VideoSort, saved, and reloaded nzbget, everything was hunky-dory. Anyways, this looks good to release to me. Thanks again.

[hugbug]

One oddity was that when I first ran it I got an error about not being able to find MoviesDir, but after I went to Settings > VideoSort, saved, and reloaded nzbget

Probably you installed this version of VideoSort into a new directory and there were no settings saved in nzbget for this new script.

Thanks for testing, I'll merge this into master branch.

[hugbug]

Merged into master. Please use the version from master-branch. It turned out the python3-branch had a few last commits missing.

[taalas]

@hugbug Awesome. Thanks for taking the time to look into this.

[brettinternet]

Thank you for making the changes in 79604c94da5c9c61966d6e55f3fd5f4db8130c1c. Should we create a new GitHub release to indicate the version increment?

[fmunteanu]

@hugbug is it possible to create a new release? For now, I use your master branch with homebrew Python 3.8 on my Mac:

$ brew install python
$ pip3 --no-cache-dir install guessit
$ pip3 show guessit
Name: guessit
Version: 3.1.1
Summary: GuessIt - a library for guessing information from video filenames.
Home-page: http://guessit.readthedocs.org/
Author: Rémi Alvergnat
Author-email: toilal.dev@gmail.com
License: LGPLv3
Location: /usr/local/lib/python3.8/site-packages
Requires: python-dateutil, rebulk, six, babelfish

$ git clone git@github.com:nzbget/VideoSort.git \
    ~/Library/Application\ Support/NZBGet/scripts/VideoSort

I noticed this warning:

VideoSort: /Users/me/Library/Application Support/NZBGet/scripts/VideoSort/lib/babelfish/converters/__init__.py:6: UserWarning: Module six was already imported from /Users/me/Library/Application Support/NZBGet/scripts/VideoSort/lib/six.py, but /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python is being added to sys.path

[BEisem]

Are there any specific version requirements for this script with Python 2? I have the script working perfectly on a machine at home, but I can't get it to work on my seedbox. I'm getting the same exact error as in the first post of this issue, but I'm not using Python 3.

I've been over the setup with my seedbox's support, and they are kind of stumped. The only difference we determined is that my home system has Python 2.7.16, but the seedbox has 2.7.13. Could this be causing a problem?

We've checked all dependencies (guessit, etc) and everything else is installed correctly. The only difference is the Python version.

[hugbug]

@BEisem This topic is about Python 3 support. For general issues please use forum https://forum.nzbget.net/viewtopic.php?f=8&t=840 or create a separate issue.

[tmm1]

:+1: for a new release with py3 support

[polizz]

I didn't see a release so I just copied your master branch. It seems to work mostly in a fresh freebsd jail with py3, however during the renaming process there is this error:

INFO    2020-12-29 01:37:23     VideoSort: Traceback (most recent call last):
INFO    2020-12-29 01:37:23     VideoSort:   File "/downloads/scripts/videosort/VideoSort.py", line 1301, in <module>
INFO    2020-12-29 01:37:23     VideoSort:     move_satellites(old_path, new_path)
INFO    2020-12-29 01:37:23     VideoSort:   File "/downloads/scripts/videosort/VideoSort.py", line 478, in move_satellites
INFO    2020-12-29 01:37:23     VideoSort:     subpart = '.' + guess['subtitle_language'][0].alpha2
INFO    2020-12-29 01:37:23     VideoSort: TypeError: 'Language' object is not subscriptable

Besides the main file that was extracted, this file was also present and I believe the culprit for this code running: 2_English.srt

I'm not a python dev so I'm not entirely sure where to start on this one. Let me know if I can provide more information. Great work updating to py3 so far, very appreciated 👍

Other than this error, it appeared that it had moved the file and renamed it properly, it just failed to cleanup.

[polizz]

Needed this fixed sooner rather than later, so I dug into the code a bit, but it's been 25 years since I've done any python, so 🤷 ... I think the array of arrays that is returned elsewhere where guessit runs, is being coerced or flattened for the `subtitle_language' property. Removing that array subscript reference and using the property directly works fine.

Also, there appeared to be an issue with the particular comparison of the base and fbase names. That code hasn't changed in years, so I don't know that I understand why it works now that I've inverted that expression - perhaps I don't understand it - it looks to me like it will never move the satellite files if they don't match the base name already.

I submitted a PR you could refer to here

[outtahere]

Using Ubuntu 20.04, nzbget 21.0, python3 and python3-guessit. VideoSort fails with the syntax error below: Seems to be a syntax problem with the octal directory permissions in pkg_resources.py causing the pp-failure but I can't figure out what?

info    Sat Apr 03 2021 22:17:29    Collection QI.S18E16.Rock.n.Roll.EXTENDED.1080p.HEVC.x265-MeGusta added to history
error   Sat Apr 03 2021 22:17:29    Post-process-script videosort/VideoSort.py for QI.S18E16.Rock.n.Roll.EXTENDED.1080p.HEVC.x265-MeGusta failed (terminated with unknown status)
info    Sat Apr 03 2021 22:17:29    VideoSort: SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
info    Sat Apr 03 2021 22:17:29    VideoSort: ^
info    Sat Apr 03 2021 22:17:29    VideoSort: def _bypass_ensure_directory(name, mode=0777):
info    Sat Apr 03 2021 22:17:29    VideoSort: File "/home/kramer/nzbget/scripts/videosort/lib/pkg_resources.py", line 51
info    Sat Apr 03 2021 22:17:29    VideoSort: from pkg_resources import iter_entry_points, EntryPoint
info    Sat Apr 03 2021 22:17:29    VideoSort: File "/home/kramer/nzbget/scripts/videosort/lib/babelfish/converters/__init__.py", line 6, in <module>
info    Sat Apr 03 2021 22:17:29    VideoSort: from .converters import (LanguageConverter, LanguageReverseConverter, LanguageEquivalenceConverter, CountryConverter,
info    Sat Apr 03 2021 22:17:29    VideoSort: File "/home/kramer/nzbget/scripts/videosort/lib/babelfish/__init__.py", line 20, in <module>

[neutralvibes]

Any chance to get a new release, as the last one was 2018, but python3 support was added 10months ago. Thanks for the work done.

[Freakadude]

Hi,

On my Qnap TS-431p nas i have Python 3.10 running (cannot downgrade it) and not sure if this is the issue but videosort does not work correctly. I get following error on post process:

VideoSort: Traceback (most recent call last): VideoSort: File "/share/Download/scripts/videosort/VideoSort.py", line 273, in VideoSort: import guessit VideoSort: File "/share/Download/scripts/videosort/lib/guessit/init.py", line 6, in VideoSort: from .api import guessit, GuessItApi VideoSort: File "/share/Download/scripts/videosort/lib/guessit/api.py", line 15, in VideoSort: from rebulk.introspector import introspect VideoSort: File "/share/Download/scripts/videosort/lib/rebulk/init.py", line 7, in VideoSort: from .rebulk import Rebulk VideoSort: File "/share/Download/scripts/videosort/lib/rebulk/rebulk.py", line 8, in VideoSort: from .match import Matches VideoSort: File "/share/Download/scripts/videosort/lib/rebulk/match.py", line 8, in VideoSort: from collections import defaultdict, MutableSequence VideoSort: ImportError: cannot import name 'MutableSequence' from 'collections' (/opt/lib/python3.10/collections/init.pyc)

I'm guessing python 3.10 fucked something up (but as mentioned i cannot downgrade). Would appreciate if someone could take a look what would fix this issue.

Thanks!

[mikenerone]

... VideoSort: from collections import defaultdict, MutableSequence VideoSort: ImportError: cannot import name 'MutableSequence' from 'collections' (/opt/lib/python3.10/collections/init.pyc) ...

I'm guessing python 3.10 fucked something up (but as mentioned i cannot downgrade). Would appreciate if someone could take a look what would fix this issue.

Hi, @Freakadude: while waiting for the next release from the real devs (I'm just some random schmo), you can fix this locally by changing:

from collections import defaultdict, MutableSequence

to

from collections import defaultdict
from collections.abc import MutableSequence

If you're curious, MutaableSequence is an abstract base class, all of which were actually moved to collections.abc in Python 3.3. It's been in a deprecation period (i.e. still usable from the old location, but it caused a deprecation warning) for about nine years, but as of 3.10, that old location was fully removed.

[Freakadude]

@mikenerone Thx for the suggestion!