Open holta opened 2 months ago
Another excerpt from the above iiab-diagnostics, just for the record!
=IIAB==========================================================================
COMMAND: cd /usr/local/calibre-web-py3; sudo git log --graph --oneline --decorate | head -50 # Calibre-Web version
* bf685667 (HEAD -> master, origin/master, origin/HEAD) Merge branch 'janeczku:master' into master
|\
| * 1eb9cd44 Improve message on error cover downlaod
| * eb656348 Fix #3153 (In Firefox, Github OAuth login goes into infinite redirect loops)
| * 2f39e51e Fix for #3151 (Invalid amazon link for uk)
| * 884f7666 Integrate advocate and change netiface requirement
| * c54e9fe5 Updated slovenian language
| |\
| | * d82944d0 Updated Slovenian translation
| |/
* | 2f7ae47e Merge branch 'janeczku:master' into master
|\|
| * 97764acb Added missing locale for datepicker in Slovenian language
| * 53f0647c New Slovenian translation
| * b5ee07eb Revert "Update Slovenian translation"
| * 5fe13f53 Update Slovenian translation
| * 89a1dcfb Fix for #3145 (improved contrast for epub reader headline)
| * 10d0542e Removed obsolete code
| * 111c93c3 Merge remote-tracking branch 'it/patch-1'
| |\
| | * 2d7cf1e6 Update italian
| * | 1d9471b2 Possible fix for #3150 (Invalid cross-device link) without coping metadata
| * | 88cb58c2 Fix for load metadata from amazon
| |/
* | 486b7bbc Merge branch 'janeczku:master' into master
|\|
| * 481237dc Fix for #3131 (Enable/Disable "Separate Book Files from Library" issue)
| * 5e9dfc07 Test update
* | 265bb088 Merge pull request #242 from holta/update_title_sort-TO-create_functions
|\ \
| * | 0035e822 cps/editbooks.py: RENAME update_title_sort TO create_functions FOR PR #240
|/ /
* | 16a1d69f Merge pull request #240 from holta/12-upstream-commits
|\ \
| * | 8c3ed8e0 12 upstream commits, 29 files changes, 2 merge conflicts resolved (cps/editbooks.py cps/uploader.py)
|/| |
| |/
| * ce720f40 Shelf sort algorithm is now saved (#3003)
| |\
| | * c92115fa Update save shelf sort algorithm
| * | a1f9f03f Fix for #3130 (only last e-mail address is saved when specifying multiple e-reader mail addresses in user settings)
| * | 6a504673 Merge branch 'Develop':
| |\|
| | * 839118e4 Bugfixes from testrun
| | * fd730f66 Bugfix generate thumbnails on gdrive
| | * 1be0ff96 Bugfix load metadata Bugfix xchange author and title Better handling of incompatible iso-639 module on python3.12
| | * 6717683a Improvements merge metadata upload format Music icon only visible once if more than one audio format available
| | * 0f9b5d08 Fix cover handling on upload formats
| | * 1eb77c9c Upload (multiple) book formats with progress and merge the corresponding metadata into the book
| | * d8d9d405 First step format upload with progress
=IIAB==========================================================================
COMMAND: /usr/bin/sudo lb --version # xklb version
2.9.058
=IIAB==========================================================================
COMMAND: /usr/bin/sudo yt-dlp --version # yt-dlp version
2024.08.06
@deldesir investigated, and FYI was unable to reproduce this during a fresh install of SMALL-sized IIAB on 64-bit Raspberry Pi OS Lite on RPi 5!
@EMG70 have you seen above that @deldesir is not able to reproduce your "502 Bad Gateway" on his RPi 5?
Can we try to dig deeper in coming days, to try to figure out what happened on your RPi 4, with another attempt?
Let's remove the "bug" label if this turns out not to be reproducible?
@EMG70 did a size ZERO-sized (unittest) and '1' (SMALL-sized) IIAB install, both failed to access Calibre-Web.
All built on a RPi 400.
He/We are starting to ask if something is arising from an OS change (Raspberry Pi OS) affecting RPi 4 & RPi 400 especially?
New IIAB size ZERO installed on a PI400 with 64-bit Raspberry Pi Lite.Cannot access IIAB home page nor calibre-web via 192.168.../books (all return 502 Bad Gateway). When recommended change is done https://github.com/iiab/iiab/discussions/3809#discussioncomment-10762627 i get 403 Forbidden error message IIAB-DIAGNOSTICS - https://paste.centos.org/view/d64c9ccc
Cannot access IIAB home page
Please remember that IIAB home page is not supposed to work when you use ZERO-sized (local_vars_unittest.yml) which contains:
admin_console_install: False
admin_console_enabled: False
Cannot access IIAB home page
Please remember that IIAB home page is not supposed to work when you use ZERO-sized (local_vars_unittest.yml) which contains:
admin_console_install: False admin_console_enabled: False
My apologies.
@deldesir please focus on fixing this, as it appears to be affecting every RPi 4, every RPi 400 (and very possibly others too!)
Hey @EMG70 and @holta , I will try to reproduce the issue on both RPI 4 and RPI 400. Will let you know...
I tried to reproduce the bug on RPI4 (for now) without success:
Test 1: Reinstalling Calibre-Web on RPI 4 using runrole --reinstall calibre-web
passed: https://paste.centos.org/view/185ad7e4
Test 2: Reinstalling Calibre-Web on RPI 4 using runrole --reinstall calibre-web
. This time the /usr/local/calibre-wep-py3
file was deleted prior running the command. https://paste.centos.org/view/13072dff
Test 3: Fresh install with Raspberry Pi OS Lite (64bit) on RPI 4: Could not proceed fully as iiab.io is not reachable for now. I used a copy of the install script but I don't have the scripts/binaries hosted at iiab.io at hand https://paste.centos.org/view/500ccacf
You don't need binaries from iiab.io during unit testing:
Start with https://raw.githubusercontent.com/iiab/iiab/refs/heads/master/vars/local_vars_unittest.yml (a.k.a. ZERO-sized) as your /etc/iiab/local_vars.yml
to keep focus on just 1 app.
Also set these in your /etc/iiab/local_vars.yml
usb_lib_install: False
usb_lib_enabled: False
Change Line 99 of /opt/iiab/iiab/roles/www_options/tasks/main.yml
to just about any working URL — instead of https://download.iiab.io/packages/heart-beat.txt
In other words Line 99 here: https://github.com/iiab/iiab/blob/4dd927cfd00c3bed1a36b9c197f554d7ee680ceb/roles/www_options/tasks/main.yml#L99
3. Change Line 99 of
/opt/iiab/iiab/roles/www_options/tasks/main.yml
to just about any working URL — instead of https://download.iiab.io/packages/heart-beat.txt In other words Line 99 here: https://github.com/iiab/iiab/blob/4dd927cfd00c3bed1a36b9c197f554d7ee680ceb/roles/www_options/tasks/main.yml#L99
Better yet, completely comment out these 20 lines — i.e. the scraping of IIAB Docs:
(e.g. in situations where FAQ.IIAB.IO and IIAB.IO are both unreachable.)
PS easiest way to do that is to set... nodocs: True
network_install: False
network_enabled: False
Issue reproduced on a fresh install on RPI 4: https://paste.centos.org/view/f177f711
@deldesir suggests the "gone missing" cryptography
module is very likely related to... piwheels or Raspberry Pi OS or similar... with these 2023 discussions being extremely relevant:
@deldesir does a modern/appropriate version of running...
pip install cryptography
...move the ball forward during testing?
Is the release of cryptography 43.0.1 on Sep 3, 2024 relevant?
2. Is the release of cryptography 43.0.1 on Sep 3, 2024 relevant?
https://pypi.org/project/cryptography/#history
Or cryptography 43.0.0 from 2024-07-20 ?
SEE: https://www.piwheels.org/project/cryptography/
Can you please manually try cryptography 42.0.8 to see if that helps? If so, reverting to 42.0.8 should be easy to hard-code later, as a temporary patch / workaround revising Calibre-web's requirements.txt file upstream... or here if necessary:
2023 discussions being extremely relevant:
@deldesir is this possibly the root cause?
@deldesir to avoid broken Calibre-Web installs like this in future, please document: (as soon as you can, here on this ticket!)
➡️ If an ugly/urgent workaround patch is necessary in the short term that's fine — but a proper analysis of what went wrong is even more important — to make very clear how we all avoid such situations in future... ! 🦅
I have just installed IIAB size 1 ON A PI 4 and calibre-web is reachable.I ran sudo apt update && sudo apt -y full-upgrade before installing IIAB SUDO IIAB-DIAGNOSTICS https://paste.centos.org/view/8d283b47 pastebinit -b paste.centos.org /var/log/apt/history.log https://paste.centos.org/view/5eca0560
Thanks @EMG70 !
@deldesir should look inside your /var/log/apt/history.log
(https://paste.centos.org/view/5eca0560) to investigate whether any very recent apt update fix[es] (e.g. python3-netifaces=0.11.0-2+b1
? or similar?) were possibly just pushed by Raspberry Pi OS for RPi 4 & 400, in the last day or two?
apt package python3-netifaces
appears irrelevant to the problem. It's unchanged since almost 2 years:
root@box:~# apt list python3-netifaces
Listing... Done
python3-netifaces/stable,now 0.11.0-2+b1 arm64 [installed]
python3-netifaces/stable 0.11.0-2+b1 armhf
root@box:~# apt changelog python3-netifaces
netifaces (0.11.0-2) unstable; urgency=medium
[ Debian Janitor ]
* Bump debhelper from old 12 to 13.
* Update standards version to 4.6.1, no changes needed.
-- Jelmer Vernooij <jelmer@debian.org> Wed, 30 Nov 2022 17:06:23 +0000
...
@deldesir (and others?) should re-test Raspberry Pi 400: try to figure out what's happening and why.
@deldesir please investigate @EMG70's very promising new theory — if this is confirmed, it should be easy for you to bisect & help all solve + fully explain the problem:
PS Does this mean that the subject line of this ticket is false, and the problem occurs on RPi 5 etc too?
Interesting. I am going to test with small-size.
A new theory is obviously needed:
@EMG70 installed an IIAB UNITTEST (in other words, Calibre-Web only) onto RPi 400 and it worked perfectly: https://paste.centos.org/view/922ad244
Please all post details (iiab-diagnostics especially) from tests, no matter the result! Onto this ticket (260) to help solve this collaboratively + quickly.
Fiddling with setting network_install: False
and network_enabled: False
is probably complicating things unnecessarily. (But if absolutely necessary, that can be tried later.)
RECAP: Keep focus on completely clean/fresh "as minimal as possible" IIAB + Calibre-Web installs — onto RPi 4, RPi 400 & RPi 5 (and VM's if necessary, for comparison) — until the hidden cause/effect pattern finally becomes obvious! :pray:
Paranoid, but... maybe try to use the exact same hardware (exact same Raspberry Pi) for subsequent/repeated tests? e.g. use the very same RPi 4 unit for every install test, or the very same RPi 400 unit (so that comparisons between many tests... can be more meaningful?)
Now I realize this is absolute paranoia (!) but yeah, do eliminate as much ambiguity as possible... testing cause/effect ~ repeatability/reproducibility in the narrowest way possible... trying to be as scientific as possible... ! 🤔
A new theory is obviously needed:
- @EMG70 installed an IIAB UNITTEST (in other words, Calibre-Web only) onto RPi 400 and it worked perfectly: https://paste.centos.org/view/922ad244
Oops I misunderstood that @EMG70 had also enabled Admin Console in the above RPi 400 test ~1.5 hours ago. In other words, it wasn't a single unit test of Calibre-Web alone 😄 ...but coincidentally is very revealing as a result!
CONCLUSION / NEW THEORY: Installing Admin Console appears to install software that (very coincidentally) allows Calibre-Web to work.
@deldesir this should hopefully be very easy to solve now! Please when you can:
cryptography
module needed by Python (anything else we need!?) and...Small-size install was successful: https://paste.centos.org/view/3ad4a9fb. Thanks Ed for pointing this out.
@deldesir for completeness, can you plz confirm that a UNITTEST install (i.e. Calibre-Web alone) fails on RPi 5?
Good idea. Will let you know.
Ansible-core installs cryptography-43.0.1. It works with python 3.12 (ubuntu 24.04) but not with python 3.11 (RPI OS which is based on debian bookworm). Calibre-Web works with small-size install on RPIs because kolibri installs python3-cryptography (version 38.0.4-3).
As expected, Calibre-Web fails with a 404 error on RPI 5 using UNNITTEST install. https://paste.centos.org/view/f71c375d
Ansible-core installs cryptography-43.0.1.
While interesting, isn't this off topic when IIAB installs ansible-core into its own venv?
Calibre-Web works with small-size install on RPIs because kolibri installs python3-cryptography (version 38.0.4-3).
Ok!
What precise package (and what precise line) of Admin Console's install adds the dependenc(ies) that allow Calibre-Web to work?
As expected, Calibre-Web fails with a 404 error on RPI 5 using UNNITTEST install. https://paste.centos.org/view/f71c375d
Why 404?
(So far @EMG70 has only mentioned "502 Bad Gateway")
In conclusion:
What is the safest and lowest-maintenance way to fix roles/calibre-web/tasks/install.yml ?
Is using Ansible to install apt package python3-cryptography
regrettably best in the end?
(Or, if there's an even lower-risk approach inside venv... that'd be even better... but perhaps that's a fantasy given recent versions of Python cryptography are all too often unavailable on Raspberry Pi OS!?)
As expected, Calibre-Web fails with a 404 error on RPI 5 using UNNITTEST install. https://paste.centos.org/view/f71c375d
As @EMG70 asks:
@deldesir don't you really want /etc/iiab/local_vars.yml to contain... ?
calibreweb_install: True
calibreweb_enabled: True
So that you end up with /etc/iiab/iiab.ini containing... ?
calibreweb_install = True
calibreweb_enabled = True
- What is the safest and lowest-maintenance way to fix roles/calibre-web/tasks/install.yml ?
Official (upstream) Calibre-Web install instructions:
FYI even the most minimal Ubuntu and Debian OS images include the apt package python3-cryptography
out-of-the-box:
root@u2410:~# aptitude why python3-cryptography
i cloud-init Depends python3-oauthlib
i A python3-oauthlib Depends python3-cryptography
root@deb12:~# aptitude why python3-cryptography
i cloud-init Depends python3-oauthlib
i A python3-oauthlib Depends python3-cryptography
root@u2204:~# aptitude why python3-cryptography
i cloud-init Depends python3-oauthlib
i A python3-oauthlib Depends python3-cryptography
Whereas this is apparently not true on Raspberry Pi OS... ?
To figure out why Calibre-Web unit testing is not yet possible on Raspberry Pi OS, @deldesir made progress with me over the past 2 hours:
apt install python3-cryptography
and apt install python3-pip
is insufficient: Calibre-Web still does not work 😢/var/log/dpkg.log
and /var/log/apt/history.log
don't seem to show much! ❓@deldesir please prepare a DRAFT (or well-tested!) PR for @EMG70 and others as soon as you solve this, so that rapid unit-testing of Calibre-Web becomes possible on Raspberry Pi OS at last, Thanks !
Should we also ask Ozzie Isaacs (i.e. the maintainer of the upstream version of Calibre-Web) to make apt install package requirements more precise — e.g. in the official instructions at https://github.com/janeczku/calibre-web/wiki/Manual-installation#manual-installation-on-linux-or-macos ?
@deldesir please try hard to explain (if you can!) why @EMG70 sees "502 Bad Gateway" whereas you see 404 — when attempting unit test installs of Calibre-Web on Raspberry Pi OS — and then visiting http://box/books (thank you !!)
Thanks @EMG70 for reporting!
Excerpt from his iiab-diagnostics (https://paste.centos.org/view/e76439c0) here: