Python3 support

[alexsalex]

Hello! I love your app, thank you for it! But Python2 is no longer supported, all new distributives get rid of the python 2. All major projects already moved to Python3, SichChill, Couchpotato etc.

When you plan to migrate the Headphones to Python3?

[rembo10]

Hey there, thanks a lot! I started doing some work on getting it working with python3. The annoying thing is that I packaged all the dependencies within the project (it was my first software project, so I was just sort of copying what sabnzbd, et all were doing). So it's a bit annoying to work through what needs to be updated.

But anyways the work will be on the python3 branch. I have some time off at the moment so hopefully I can get it working soon.

Sorry for the delay in replying to this btw, since it's quite an important issue

[rembo10]

Let's use this issue to track bugs in the migration. As of right now:

• I've disabled twitter notifications because I want to use the requests oauth lib instead of the one we were using, which I think has been abandoned
• Searches are using bytestrings, so they're not getting any matches

[alexsalex]

If you need any help, please let us know. I would like to help. DM me if interested.

[rembo10]

That would be very much appreciated. Basically everything needs to be tested... I'll try to do as much as I can on my own but there are some things I can't test.... so I don't know the best way to solve this. Maybe just get a mostly functioning version into master and then fix the bugs as they come up....

[rembo10]

Searches are working now (it was a problem with the old feedparser dependency) and I've disabled twitter notifications, because it seems like it wasn't working (the existing key/secret was giving me a 401 error, and the flow doesn't seem to work properly even on master.)

[rembo10]

@alexsalex btw, I can't seem to find the python3 couchpotato?

[rembo10]

The python3 branch has been tested working with the headphones indexer and sabnzbd, and the following postprocessor options:

• Moving/renaming files
• Embedding album art
• Embedding lyrics
• Correcting metadata
• Re-encoding
• Pushover notifications

So I think the basic functionality is there. I think all the file encoding stuff might bite us again (python3 now uses unicode strings) but we can burn that bridge when we get there.

If anyone wants to try it, I'd probably recommend backing up your database and config beforehand, just in case.

[alexsalex]

I would like to try it. Regarding the CouchPotato, let me check.

[rembo10]

The python3 branch is already on github. Definitely let me know if you run into bugs... I think if I can get a few more reports that it's ok and doesn't do anything to the database/config, I can push it into develop and hopefully get more testers. But definitely backup config/db just in case

[rembo10]

python3 branch has been merged into develop

[JMSDOnline]

@rembo10 thanks for working on getting python3 support in here! I've grabbed the develop branch and the only issue I encountered from the initial install was the following:

1. Seems we need a zone = ascii(zone).decode(utf8) here: https://github.com/rembo10/headphones/blob/master/lib/pytz/__init__.py#L166

2. Had to adjust file( to open( here: https://github.com/rembo10/headphones/blob/develop/headphones/__init__.py#L238 and here: https://github.com/rembo10/headphones/blob/develop/headphones/__init__.py#L254

   Once I made these adjustments I was able to start the service on Debian 11 without the TypeError: a bytes-like object is required, not 'str'

[rembo10]

Ah, I didn't try to run it daemonized.... I'll get a fix up tomorrow.

I think the pytz lib can be updated to the latest version, I think it just went through the py2to3 process. The other two should have the changes you suggested. I don't know why the 2to3 tool didn't catch it.

Thanks so much for the report.

[rembo10]

Nevermind, the fix is up now. ~~Also putting this here as a reminder to myself: Need to unidecode the folder name for sab, since it does it's own renaming which headphones then won't pick up to automatically post process~~

[JMSDOnline]

Can confirm, seems all set now!

[mgaulton]

Ran into a snag today when my service wouldn't start. git pull from master. Any suggestions?

Python is 3.8.12 on Opensuse Tumbleweed

Running: sudo find . -name "*.pyc" -exec rm -f {} \; as suggested in some googling, results in a 500 error on http://host:8181/config

500 Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.

Traceback (most recent call last):
  File "/opt/headphones-master/lib/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/opt/headphones-master/lib/cherrypy/_cprequest.py", line 697, in _do_respond
    response.body = self.handler()
  File "/opt/headphones-master/lib/cherrypy/lib/encoding.py", line 223, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/opt/headphones-master/lib/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/opt/headphones-master/headphones/webserve.py", line 1437, in config
    extras = list(map(int, headphones.CONFIG.EXTRAS.split(',')))
ValueError: invalid literal for int() with base 10: '"1'
Powered by CherryPy 18.6.1

erver6:/opt/headphones-master # ./Headphones.py Headphones requires Python >= 3.5 server6:/opt/headphones-master # python3 ./Headphones.py Traceback (most recent call last): File "./Headphones.py", line 27, in from headphones import webstart, logger File "/opt/headphones-master/headphones/init.py", line 20, in import subprocess File "/usr/lib64/python3.8/subprocess.py", line 49, in import signal File "/usr/lib64/python3.8/signal.py", line 4, in from enum import IntEnum as _IntEnum ImportError: bad magic number in 'enum': b'\x03\xf3\r\n'

[oztrich]

Headphones prompted me to update tonight, but then would not start afterwards (running as a service). If I try running manually I get this:

server:/opt/headphones$ python Headphones.py Headphones requires Python >= 3.5

If I try running it using Python 3 instead: server:/opt/headphones$ python3 Headphones.py /opt/headphones/lib/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: 0.1.39ubuntu1 is an invalid version and will not be supported in a future release warnings.warn( /opt/headphones/lib/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: 1.14.0-unknown is an invalid version and will not be supported in a future release warnings.warn( Traceback (most recent call last): File "/opt/headphones/Headphones.py", line 27, in from headphones import webstart, logger File "/opt/headphones/headphones/init.py", line 30, in from headphones import versioncheck, logger File "/opt/headphones/headphones/versioncheck.py", line 23, in from headphones import logger, version, request File "/opt/headphones/headphones/request.py", line 21, in import requests File "/opt/headphones/lib/requests/init.py", line 134, in from . import packages ImportError: bad magic number in 'requests.packages': b'\x03\xf3\r\n'

[rembo10]

@oztrich try removing the .pyc files (`find . -name '.pyc' -delete` in the headphones dir)

@mgaulton: can you try (with headphones shut down) editing your config, to remove any quotes from the extras setting? It's using the native config parsing library now, which will read every option as a string. I can try to put in a check but this will be the quickest fix

Thanks for the reports, i didn't test it with the EXTRAS set... Not sure the best way to handle the pyc problem....

[oztrich]

@oztrich try removing the .pyc files (`find . -name '.pyc' -delete` in the headphones dir)

This worked!

After removing the .pyc files I could start Headphones calling it using python3.

For my .service I amended the start from: ExecStart=/opt/headphones/Headphones.py --config /opt/headphones/config.ini --datadir /opt/headphones to: ExecStart=python3 /opt/headphones/Headphones.py --config /opt/headphones/config.ini --datadir /opt/headphones

All seems to be working ok now. Thanks!

[mgaulton]

@rembo10 that works, config loads properly now. thank you.

as to to the pyc, I just checked and I remembered right, nzbToMedia does a cleanup on start. Perhaps you could look at that?

-- Cleaning bytecode -- Removing cleanup.pyc Removing eol.pyc

Removing cleanup.pyc Removing eol.pyc

[rembo10]

@mgaulton Thanks for the heads up.... I'm looking into it

[jrc27]

I'm scratching my head here. The update killed me too. I tried the delete of the .pyc files and that didn't work. I can't even get a brand new install to start up. I'm sure part/all of the problem is that Sickchill is using it's own version of Python3 and my couchpotato and lazylibrarian only want to work with Python2.7. No idea what to do at this point.

[rembo10]

What do you mean "sickchill is using it's own version of python 3"? What system are you running? How are you launching headphones? What kind of error do you get on startup?

[jrc27]

i'm on windows. i was trying to launch by double clicking the .py file. My meaning of Sickchill was that it installed python3 as a subfolder of the sickchill folder. I'll be honest, i know just enough about this stuff to get it up and running and that's about it. When i was trying to launch headphones i wasn't getting anything. i downloaded the last phython 2.7 version and copied over my database and config file and it runs fine. Worst case, i just live with running an outdated version.

[rembo10]

Gotcha. Yeah you can stick with the older version, it's just that some Linux distributions might stop packaging Python 2 since it's end-of-life.

How were you trying to launch headphones though? With python 3? And the latest zip from GitHub? And you were launching it from a terminal or? It would be cool to try to resolve this for future users but if you're content on the older version not to worry

[raymondjstone]

You should be able to run headphones by running it using python 3.10 by starting it with the correct python.exe ie C:\Python310\phython,.exe headphones.py

When trying to update headphones I had lot of issues, I eventually had to delete my headphones folder totally, install the new version and the copy in the database and config.ini file to the correct location (which was not for whatever reason the same place I had it originally), it then started to work fine

[jrc27]

My sequence of events as i recall them (i'm old) lol. I ran the updater and it wouldn't work, mind you at this time the only version of python3 i had was in the Sickchill folder and i'm not even sure what version it is. So, i'm not sure if headphones was trying to still use the 2.7 or not. I had headphones running as a service and i deleted it after i couldn't get it to run. I was trying to launch it afterwords by double clicking the headphones.py file in the headphones folder. After that didn't work, i donloaded and extracted the lastet zip, installed the latest version of python3 and specifically pointed the headphones.py to that python3 location and it wouldn't launch. I'm more than willing to help troubleshoot, now that i know i can always get it back running using 2.7.

Thanks for the help fellas, it really is appreciated and so is your efforts with this project!!

[rembo10]

All good. Try it from the command prompt... I'm not exactly sure how on windows, something like: > C:\path\to\python3.exe C:\path\to\Headphones.py

[jrc27]

PS C:\Python36> python3.6.exe e:\Headphones\Headphones.py Traceback (most recent call last): File "e:\Headphones\lib\jaraco\text__init__.py", line 7, in from importlib.resources import files # type: ignore ModuleNotFoundError: No module named 'importlib.resources'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "e:\Headphones\Headphones.py", line 27, in from headphones import webstart, logger File "e:\Headphones\headphones__init.py", line 27, in import cherrypy File "e:\Headphones\lib\cherrypy__init__.py", line 66, in from ._cperror import ( File "e:\Headphones\lib\cherrypy_cperror.py", line 135, in from cherrypy.lib import httputil as _httputil File "e:\Headphones\lib\cherrypy\lib\httputil.py", line 20, in import jaraco.collections File "e:\Headphones\lib\jaraco\collections.py", line 10, in import jaraco.text File "e:\Headphones\lib\jaraco\text\init__.py", line 9, in from importlib_resources import files # type: ignore ModuleNotFoundError: No module named 'importlib_resources'

[Blackbear199]

i also got this error but on my nas(python 3.8.2) i tried it on windows(python 3.10.2) and it works fine.

[Blackbear199]

think i figured it out.

importlib.resources isnt included in python3 according to here it wasnt included until version 3.9+? https://pypi.org/project/importlib-resources/

[rembo10]

I thought it was included in 3.7, which was why I couldn't understand why it was failing on 3.8+. But maybe the api is different... I'll package the shim with the libs. Thanks for the find.

[Blackbear199]

i see you added importlib resources

i still get the same initial err as posted above in the During handling of the above exception, another exception occurred:

i now see this added as the last entry File "/raid/data/module/Headphones/bin/headphones/lib/importlib_resources/abc.py", line 4, in from ._compat import runtime_checkable, Protocol File "/raid/data/module/Headphones/bin/headphones/lib/importlib_resources/_compat.py", line 11, in from zipp import Path as ZipPath # type: ignore ModuleNotFoundError: No module named 'zipp'

looks like need another module added.

[rembo10]

Sorry about that, I missed it in the importlib_resources setup.cfg. Fixed in master/develop

[Blackbear199]

good job. working now on my nas python 3.8.2

[Blackbear199]

heres some more errors from log.

this one shows up when a force check is done.

2022-02-14 11:25:34 | ERROR | Uncaught exception: Traceback (most recent call last):File "/raiddata/0/module/Headphones/bin/headphones/headphones/logger.py", line 215, in new_runold_run(*args, *kwargs)File "/raid/data/module/Python3/sys/lib/python3.8/threading.py", line 870, in runself._target(self._args, **self._kwargs)File "/raiddata/0/module/Headphones/bin/headphones/headphones/searcher.py", line 221, in searchforalbumdo_sorted_search(album, new, losslessOnly)File "/raiddata/0/module/Headphones/bin/headphones/headphones/searcher.py", line 333, in do_sorted_searchsend_to_downloader(data, bestqual, album)File "/raiddata/0/module/Headphones/bin/headphones/headphones/searcher.py", line 1006, in send_to_downloadertorrentid = calculate_torrent_hash(bestqual[2], data)File "/raiddata/0/module/Headphones/bin/headphones/headphones/searcher.py", line 139, in calculate_torrent_hashinfo = bdecode(data)["info"]KeyError: 'info'

this one when a torrent is sent to qbittorent.

2022-02-14 11:25:32 | ERROR | Request raised exception: Invalid URL '': No scheme supplied. Perhaps you meant http://?

qbittorent still recieves the torrent and downloads it. most likely python3 related,why i posted it here.

[Blackbear199]

lookin at the log more i think the second error above has something more to do with handling search results,not qborrent. heres more of the log

2022-02-14 13:30:15 | INFO | Sending torrent to QBiTorrent 2022-02-14 13:30:15 | INFO | Found best result from Jackett_1337x: xxxxxx Mp3 320kbps Hunter - 115.9 MB 2022-02-14 13:30:13 | INFO | Making sure we can download the best result 2022-02-14 13:30:13 | ERROR | Request raised exception: Invalid URL '': No scheme supplied. Perhaps you meant http://? 2022-02-14 13:30:13 | INFO | Parsing results from using search term: xxxxxxxxxx

[rembo10]

I think I have a fix for the KeyError, it's in the develop branch. I couldn't replicate the invalid url error. What do you have set for the jackett host? or qbittorrent host?

[Blackbear199]

jackett host is localhost qbitorrent host is lan ip to my pc as all my downloaders run on it with vpn

[rembo10]

No but i mean specifically what are the URLs you have there? It might be better for this to be a new issue. Because I could reproduce the KeyError, which is fixed in develop, which I thought might be causing the invalid url error somehow, but i couldn't reproduce it

[Blackbear199]

updated to develop and logs r clean of all errors now. quick,quest.how can i disable the update windows in bottom right that pops up.

[rembo10]

Cool, thanks for being the guinea pig. I think I'll close this issue for now and anything new just open up a new specific issue..... cheers