search

DobyTang / LazyLibrarian

This project isn't finished yet. Goal is to create a SickBeard, CouchPotato, Headphones-like application for ebooks. Headphones is used as a base, so there are still a lot of references to it.
732 stars 70 forks source link

Missing Modules? #1457

Closed famewolf closed 6 years ago

famewolf commented 6 years ago

To help with identifying and fixing issues, please include as much information as possible, including:

LazyLibrarian version number (at the bottom of config page)

Operating system used (windows, mac, linux, NAS type)

Interface in use (default, bookstrap)

Which api (Goodreads, GoogleBooks, both)

Source of your LazyLibrarian installation (git, zip file, 3rd party package)

Relevant debug log with api keys and any passwords redacted

Please note - usually a single line of log is not sufficient. The lines just before the error occurs can give useful context and greatly assist with debugging.

There is a built-in debug log creator on the logs page which makes it easy to provide this information

I have lazylibrarian running on freenas using the pre-compiled package. I've been doing the normal updates for a long time now..when I went into it this time it told me it could not get the latest git and when I browsed the log I saw:

2018-06-11 03:00:23 WARNING (getLatestVersion_FromGit) Could not get the latest commit from github
2018-06-11 03:00:22 WARNING cryptography: module missing
2018-06-11 03:00:22 WARNING OpenSSL missing module/attribute: No module named OpenSSL.SSL
2018-06-11 03:00:22 WARNING pyOpenSSL: module missing
2018-06-11 03:10:29 ERROR Unhandled exception in processDir: Traceback (most recent call last): File "/usr/pbi/lazylibrarian-amd64/share/lazylibrarian/LazyLibrarian/lazylibrarian/postprocess.py", line 455, in processDir res = unpack_archive(os.path.join(pp_path, f), pp_path, matchtitle) File "/usr/pbi/lazylibrarian-amd64/share/lazylibrarian/LazyLibrarian/lazylibrarian/postprocess.py", line 197, in unpack_archive if not os.path.isfile(pp_path): # regular files only File "/usr/pbi/lazylibrarian-amd64/lib/python2.7/genericpath.py", line 29, in isfile st = os.stat(path) UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 145: ordinal not in range(128)
-- -- --

Obviously I'm concerned. Is there anything I can do to get things back on track or am I going to have to "start fresh?"

philborman commented 6 years ago

Couldn't get the latest git is usually a communication issue, often temporary. If it continues, turn debug log on and use the "check version" button in config. That should report what the problem is.

The missing modules are only a warning, see the lazylibrarian wiki page on SSL Potentially could be a problem in the future if goodreads only allow https, which was announced last December but doesn't seem to have happened yet.

The unhandled exception in processdir needs fixing though.

philborman commented 6 years ago

The unhandled exception should be fixed in the next release, probably tomorrow.

famewolf commented 6 years ago

Thanks for the quick followup! I'll have to enable debugging to figure out why it's still "Unable to get latest version from GITHub". I verified no proxy was set that could be interfering. It's still pulling it author data so I know it has internet..so just the upgrade part is having problems.

philborman commented 6 years ago

Any news on the upgrade issue?

famewolf commented 6 years ago

screenshot_20180630_144243

I enabled debug mode and here's the portion of log: Timestamp Level Message
2018-06-30 17:26:19 INFO Debug log OFF, loglevel is 1
2018-06-30 17:25:56 INFO Debug log saved as /var/db/lazylibrarian/Logs/debug.zip
2018-06-30 17:25:56 DEBUG 394 log lines written to /var/db/lazylibrarian/Logs/debug.log
2018-06-30 17:25:56 DEBUG Redacted 9 passwords/apikeys
2018-06-30 17:25:56 DEBUG Found "Debug log ON" line 4 in /var/db/lazylibrarian/Logs/lazylibrarian.log
2018-06-30 17:25:56 DEBUG Processing logfile [/var/db/lazylibrarian/Logs/lazylibrarian.log]
2018-06-30 17:25:50 INFO Debug log saved as /var/db/lazylibrarian/Logs/debug.zip
2018-06-30 17:25:50 DEBUG 402 log lines written to /var/db/lazylibrarian/Logs/debug.log
2018-06-30 17:25:50 DEBUG Redacted 9 passwords/apikeys
2018-06-30 17:25:50 DEBUG Found "Debug log ON" line 12 in /var/db/lazylibrarian/Logs/lazylibrarian.log
2018-06-30 17:25:50 DEBUG Processing logfile [/var/db/lazylibrarian/Logs/lazylibrarian.log]
2018-06-30 17:25:29 DEBUG Update check complete
2018-06-30 17:25:29 DEBUG (getCommitDifferenceFromGit) - exiting with commit value of [0]
2018-06-30 17:25:29 INFO Unable to get latest version from GitHub
2018-06-30 17:25:29 DEBUG git SSLError for https://api.github.com/repos/dobytang/lazylibrarian/commits/master: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:581)
2018-06-30 17:25:29 WARNING (getLatestVersion_FromGit) Could not get the latest commit from github
2018-06-30 17:25:29 DEBUG (getLatestVersion_FromGit) Checking if modified since Sat, 10 Feb 2018 19:29:11 GMT
2018-06-30 17:25:29 DEBUG (getLatestVersion_FromGit) Retrieving latest version information from github command=[https://api.github.com/repos/dobytang/lazylibrarian/commits/master]
2018-06-30 17:25:29 DEBUG (getCurrentVersion) - Install type [source] Local Version is unchanged [b53cbb15f05265e21c022a4d9c20310b8a6949b7]
2018-06-30 17:25:29 DEBUG (updateVersionFile) Updating [/usr/pbi/lazylibrarian-amd64/share/lazylibrarian/LazyLibrarian/version.txt] with value [b53cbb15f05265e21c022a4d9c20310b8a6949b7]
2018-06-30 17:25:29 DEBUG (getInstallType) [source] install detected. Setting Branch to [master]
2018-06-30 17:25:29 DEBUG Set Install Type, Current & Latest Version and Commit status
2018-06-30 17:25:00 INFO Debug log ON, loglevel is 2
philborman commented 6 years ago

Enable debugging, level 2 should be enough, and try the "check version" button on the config page, should show what's going on. Also maybe try with bookstrap interface in case it's related to the legacy one which doesn't get much testing these days. Shouldn't be that, but you never know...

famewolf commented 6 years ago

Previous comment updated with correct portion of the log with check update done. Not familiar with "bookstrap interface" so you'd have to point me in right direction.

philborman commented 6 years ago

The problem in the log is SSL error Will need to look into it and get back to you. What operating system are you running lazylibrarian on?

Bookstrap interface is probably not relevant to this, but it's a much more modern interface and many of the new features in lazylibrarian are only in the new interface. Select it from the dropdown box on the first page of lazylibrarian config.

famewolf commented 6 years ago

I am running it on a Freenas server using a precompiled/assembled plugin in a jail. Freenas is based on freebsd. The original install was quite some time back and then the package has been self updating itself since.

Bookstrap looks awesome. Not sure how I missed that all this time.

philborman commented 6 years ago

I don't think there are an recent changes to the ssl section in lazylibrarian, but there were some changes 3 months ago, and some more 6 months ago that might be relevant. What version of lazylibrarian are you running (so we know when the update last worked). Would narrow down where to look. update scrap that, you are using a version b53cbb1 from February, it says earlier in this thread.

Now you have the bookstrap interface running can you try the "System Info" button on the config page (bottom of the interface tab). What version of "requests" does it say in the popup, and anything in there about ssl?

A quick google of the ssl error message throws up

it is because of the recent TLS deprecation by Python.org sites

Do you have access to a commandline on your freenas, and does it have pip ? if so it might be as simple as pip install requests pip install 'requests[security]'

famewolf commented 6 years ago

Repo: https://github.com/dobytang/lazylibrarian : Branch: master : Updated: Sat Feb 10 14:29:11 2018 Current Version: b53cbb15f05265e21c022a4d9c20310b8a6949b7 : Latest Version: Not_Available_From_GitHUB screenshot_20180701_032052 screenshot_20180701_032154 screenshot_20180701_032244 screenshot_20180701_032347 screenshot_20180701_032435 screenshot_20180701_032529

philborman commented 6 years ago

Sorry, just updating my previous reply as you were sending this, see the new reply :-)

famewolf commented 6 years ago

The TLS issue seems to be a vicious circle. I installed pip into the jail and tried running the commands you indicated:

pip install requests

Collecting requests Could not fetch URL https://pypi.python.org/simple/requests/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping Could not find a version that satisfies the requirement requests (from versions: ) No matching distribution found for requests

philborman commented 6 years ago

https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html

Need to figure out how to get around this. Needs some thought...

famewolf commented 6 years ago

As a last ditch effort I can look into upgrading the jumpdrive the nas boots off of to the latest version. I'm currently on 9.10 with the latest being 11.1 however my usual opinion is "if it ain't broke don't fix it." All other plugins such as sabnzbd, sickrage, couchpotato, nzbhydra etc along with the 20TB of storage continue to work as expected so not in a hurry to possibly screw that up.

I do appreciate all the effort on your part however. I did a full pkg upgrade of the lazylibrarian jail but it didn't change anything regarding the error pip gave.

philborman commented 6 years ago

I agree, major upgrades are best avoided unless essential, and no guarantee it would fix this particular issue anyway. The problem seems to be the old version of python used in the freenas os. Later versions of python include tls 1.2 by default. Might be possible to upgrade the version of python used in the lazylibrarian jail?

Would be simpler to periodically upgrade lazylibrarian from a zipfile, but you will hit other issues if more sites disable tls 1.0 eg goodreads or some of your torrent providers

Had a quick look at sickrage (sickbeard/couchpotato/headphones/lazylibrarian all started from a common code base years ago) and it uses http to talk to git for updates, not https like we do, but git have talked about forcing https access for a while now. nzbhydra uses https though, wonder how they get around it, or do updates not work with that either?

philborman commented 6 years ago

Another thing you could try, purely as a test, not a proper fix... Edit lazylibrarian/versioncheck.py and change all occurrences of https:// to http:// If that lets you past the error the upgrade will still not work (it will complain there are local changes) but I can set the code to try https first and then http if https fails. Might work?

famewolf commented 6 years ago

I changed it to http as requested.....how's this for a path: /usr/local/share/lazylibrarian/LazyLibrarian/lazylibrarian

Timestamp Level Message
2018-07-01 12:04:09 INFO Debug log OFF, loglevel is 1
2018-07-01 12:03:40 DEBUG Update check complete
2018-07-01 12:03:40 DEBUG (getCommitDifferenceFromGit) - exiting with commit value of [0]
2018-07-01 12:03:40 INFO Unable to get latest version from GitHub
2018-07-01 12:03:40 DEBUG git SSLError for http://api.github.com/repos/dobytang/lazylibrarian/commits/master: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:581)
2018-07-01 12:03:40 WARNING (getLatestVersion_FromGit) Could not get the latest commit from github
2018-07-01 12:03:39 DEBUG (getLatestVersion_FromGit) Checking if modified since Sat, 10 Feb 2018 19:29:11 GMT
2018-07-01 12:03:39 DEBUG (getLatestVersion_FromGit) Retrieving latest version information from github command=[http://api.github.com/repos/dobytang/lazylibrarian/commits/master]
2018-07-01 12:03:39 DEBUG (getCurrentVersion) - Install type [source] Local Version is unchanged [b53cbb15f05265e21c022a4d9c20310b8a6949b7]
2018-07-01 12:03:39 DEBUG (updateVersionFile) Updating [/usr/pbi/lazylibrarian-amd64/share/lazylibrarian/LazyLibrarian/version.txt] with value [b53cbb15f05265e21c022a4d9c20310b8a6949b7]
2018-07-01 12:03:39 DEBUG (getInstallType) [source] install detected. Setting Branch to [master]
2018-07-01 12:03:39 DEBUG Set Install Type, Current & Latest Version and Commit status
2018-07-01 12:03:28 INFO Debug log ON, loglevel is 2
philborman commented 6 years ago

I'm guessing git redirects you to https. Oh well, it was worth a try. One final thing you could try, if you don't mind, in the same versioncheck file near the top is a block 

try:
    import requests
except ImportError:
    import lib.requests as requests

If you swap those over so the first try is import lib.requests as requests It will use a "local" copy of requests instead of the "jail" copy, does it make any difference?

famewolf commented 6 years ago

Did you intend me to switch back to https when the swap was done because I didn't. Swapped them out and after getting the indentation right so it restarted got the following log:

Timestamp Level Thread File Method Line No Message
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 Update check complete
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 (getCommitDifferenceFromGit) - exiting with commit value of [0]
2018-07-01 20:01:39 INFO UPDATES versioncheck.py logmsg 43 Unable to get latest version from GitHub
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 git SSLError for http://api.github.com/repos/dobytang/lazylibrarian/commits/master: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:581)
2018-07-01 20:01:39 WARNING UPDATES versioncheck.py logmsg 47 (getLatestVersion_FromGit) Could not get the latest commit from github
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 (getLatestVersion_FromGit) Checking if modified since Sat, 10 Feb 2018 19:29:11 GMT
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 (getLatestVersion_FromGit) Retrieving latest version information from github command=[http://api.github.com/repos/dobytang/lazylibrarian/commits/master]
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 (getCurrentVersion) - Install type [source] Local Version is unchanged [b53cbb15f05265e21c022a4d9c20310b8a6949b7]
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 (updateVersionFile) Updating [/usr/pbi/lazylibrarian-amd64/share/lazylibrarian/LazyLibrarian/version.txt] with value [b53cbb15f05265e21c022a4d9c20310b8a6949b7]
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 (getInstallType) [source] install detected. Setting Branch to [master]
2018-07-01 20:01:39 DEBUG UPDATES versioncheck.py logmsg 45 Set Install Type, Current & Latest Version and Commit status
2018-07-01 20:01:32 INFO WEBSERVER webServe.py toggleLog 2908 Debug log ON, loglevel is 2
famewolf commented 6 years ago

Oh I noticed nzbhydra was using an http proxy so I tried the same proxy in lazylibrarian using both the original versioncheck.py and the modded one but didn't seem to change anything. I did NOT check the "enable http proxy" box because for some reason I recall in a previous conversation that didn't work the way I thought it did so this was perhaps a useless test.

The proxy runs in the same jail as transmission which directs all torrent traffic through a vpn so the proxy is also "anonymized" to that extent for the nzbhydra http traffic.

philborman commented 6 years ago

Ok, thanks for trying. Seems the local requests library still uses the same version of TLS. Switching back to http doesn't matter for the test as git redirects http calls to https. The "enable http proxy" box depends on what proxy you are using. Needs to be left unchecked for nginx. FWIW the latest lazylibrarian shows the tls version in the system info popup, but I think it will only confirm what we already know.

I think I might need to go through nzbhydra sources to see how they do it

philborman commented 6 years ago

https://forums.freenas.org/index.php?threads/help-updating-openssl-in-a-freenas-9-3-jail.63285/ looks like the same problem, seems we are not going to solve this without upgrading freenas

nzbhydra doesn't do anything special, just uses the system version of requests, so presumably it doesn't need tls > 1.0 for the sites it's talking to. I will close this as it's not something lazylibrarian can work around, but let me know if you get it fixed and we can put something in the wiki to help others with the same problem. Thanks.