oerdnj / deb.sury.org

Public bugreports for anything ppa:ondrej/*
800 stars 26 forks source link

Apt fails to complete an NGINX upgrade, NGINX left in failed state #2119

Closed ComputerTinker closed 2 months ago

ComputerTinker commented 2 months ago

Frequently asked questions

Describe the bug

Apt fails to complete an NGINX upgrade, NGINX left in failed state

What steps did you take to resolve issue yourself before reporting it here

Here are the NGINX packages I had installed initially:

libnginx-mod-brotli/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6]
libnginx-mod-http-geoip/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
libnginx-mod-http-image-filter/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
libnginx-mod-http-xslt-filter/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
libnginx-mod-mail/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
libnginx-mod-ssl-ct/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1:1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6]
libnginx-mod-stream-geoip/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
libnginx-mod-stream/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
nginx-common/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 all [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
nginx-core/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
nginx/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 all [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]

I made sure to do an apt update to get the latest packages. Then if I just do an 'apt upgrade' it threatens to remove several NGINX packages:

sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libnginx-mod-brotli libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-ssl-ct libnginx-mod-stream
  libnginx-mod-stream-geoip nginx-core
Use 'sudo apt autoremove' to remove them.
Get more security updates through Ubuntu Pro with 'esm-apps' enabled:
  python2.7-minimal redis-server libmagickwand-6.q16-6 redis-tools
  libmagickcore-6.q16-6 imagemagick-6-common python2.7 libpython2.7-minimal
  libpython2.7-stdlib
Learn more about Ubuntu Pro on AWS at https://ubuntu.com/aws/pro
The following NEW packages will be installed:
  debsuryorg-archive-keyring libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static ubuntu-pro-client
The following packages have been kept back:
  python3-update-manager update-manager-core
The following packages will be upgraded:
  coreutils ethtool libc-bin libc-dev-bin libc-devtools libc6 libc6-dev libldap-2.5-0 libldap-common libnginx-mod-brotli libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-ssl-ct
  libnginx-mod-stream libnginx-mod-stream-geoip libpython2.7-minimal libpython2.7-stdlib locales nginx nginx-common nginx-core nodejs openssh-client openssh-server openssh-sftp-server php8.2-bcmath php8.2-cli php8.2-common php8.2-curl php8.2-fpm
  php8.2-gd php8.2-gmp php8.2-intl php8.2-mbstring php8.2-mysql php8.2-opcache php8.2-readline php8.2-sqlite3 php8.2-xml php8.2-zip php8.3-cli php8.3-common php8.3-opcache php8.3-phpdbg php8.3-readline python2.7 python2.7-minimal snapd
  ubuntu-advantage-tools ubuntu-core-launcher ubuntu-pro-client-l10n update-notifier-common
54 upgraded, 4 newly installed, 0 to remove and 2 not upgraded.
6 standard LTS security updates
Need to get 86.4 MB of archives.
After this operation, 1,159 kB disk space will be freed.
Do you want to continue? [Y/n] n

I tried explicitly specifying that several of the NGINX packages be installed, so that the only thing which would be uninstalled would be the libnginx-mod-brotli package. (I saw in your other post has been replaced by the separate -filter and -static packages) When I proceeded with the upgrade, however, it failed to complete properly:

sudo apt install nginx-core libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static libnginx-mod-ssl-ct
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  libnginx-mod-brotli
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  debsuryorg-archive-keyring libnginx-mod-brotli libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libnginx-mod-stream-geoip nginx nginx-common
Suggested packages:
  fcgiwrap nginx-doc ssl-cert
The following NEW packages will be installed:
  debsuryorg-archive-keyring libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static
The following packages will be upgraded:
  libnginx-mod-brotli libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-ssl-ct libnginx-mod-stream libnginx-mod-stream-geoip nginx nginx-common nginx-core
11 upgraded, 3 newly installed, 0 to remove and 45 not upgraded.
Need to get 1,101 kB of archives.
After this operation, 99.3 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 debsuryorg-archive-keyring all 2024.02.05+ubuntu22.04.1+deb.sury.org+1 [4,326 B]
Get:2 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-stream amd64 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [108 kB]
Get:3 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-stream-geoip amd64 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [37.6 kB]
Get:4 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-ssl-ct amd64 1:1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6 [7,844 B]
Get:5 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-mail amd64 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [76.0 kB]
Get:6 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-http-xslt-filter amd64 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [40.9 kB]
Get:7 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-http-image-filter amd64 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [42.5 kB]
Get:8 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 nginx-common all 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [64.0 kB]
Get:9 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-http-brotli-filter amd64 1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6 [9,648 B]
Get:10 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-http-brotli-static amd64 1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6 [7,588 B]
Get:11 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-brotli all 1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6 [4,118 B]
Get:12 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 nginx-core all 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [33.1 kB]
Get:13 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 nginx amd64 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [627 kB]
Get:14 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 libnginx-mod-http-geoip amd64 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 [38.5 kB]
Fetched 1,101 kB in 2s (481 kB/s)
Preconfiguring packages ...
Selecting previously unselected package debsuryorg-archive-keyring.
(Reading database ... 117013 files and directories currently installed.)
Preparing to unpack .../00-debsuryorg-archive-keyring_2024.02.05+ubuntu22.04.1+deb.sury.org+1_all.deb ...
Unpacking debsuryorg-archive-keyring (2024.02.05+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../01-libnginx-mod-stream_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libnginx-mod-stream (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../02-libnginx-mod-stream-geoip_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libnginx-mod-stream-geoip (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../03-libnginx-mod-ssl-ct_1%3a1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6_amd64.deb ...
Unpacking libnginx-mod-ssl-ct (1:1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../04-libnginx-mod-mail_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libnginx-mod-mail (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../05-libnginx-mod-http-xslt-filter_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libnginx-mod-http-xslt-filter (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../06-libnginx-mod-http-image-filter_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libnginx-mod-http-image-filter (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../07-nginx-common_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_all.deb ...
Unpacking nginx-common (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
dpkg: warning: unable to delete old directory '/var/www/html': Directory not empty
dpkg: warning: unable to delete old directory '/var/www': Directory not empty
Selecting previously unselected package libnginx-mod-http-brotli-filter.
dpkg: considering deconfiguration of libnginx-mod-brotli, which would be broken by installation of libnginx-mod-http-brotli-filter ...
dpkg: yes, will deconfigure libnginx-mod-brotli (broken by libnginx-mod-http-brotli-filter)
Preparing to unpack .../08-libnginx-mod-http-brotli-filter_1%3a1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6_amd64.deb ...
De-configuring libnginx-mod-brotli (1.25.3-1+ubuntu22.04.1+deb.sury.org+1), to allow installation of libnginx-mod-http-brotli-filter (1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6) ...
Unpacking libnginx-mod-http-brotli-filter (1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6) ...
Replacing files in old package libnginx-mod-brotli (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../09-nginx-core_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_all.deb ...
Unpacking nginx-core (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../10-nginx_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.3-1+ubuntu22.04.1+deb.sury.org+1) ...
Setting up debsuryorg-archive-keyring (2024.02.05+ubuntu22.04.1+deb.sury.org+1) ...
Setting up nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
 * Upgrading binary nginx                                                                                                                                                                                                                              [fail]
invoke-rc.d: initscript nginx, action "upgrade" failed.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
invoke-rc.d: initscript nginx, action "restart" failed.
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-04-18 14:53:51 EDT; 10ms ago
       Docs: man:nginx(8)
    Process: 160910 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 7ms

Apr 18 14:53:51 onelink5 systemd[1]: Starting A high performance web server and a reverse proxy server...
Apr 18 14:53:51 onelink5 nginx[160910]: 2024/04/18 14:53:51 [emerg] 160910#160910: module "/usr/share/nginx/modules/ngx_http_geoip_module.so" version 1025003 instead of 1025005 in /etc/nginx/modules-enabled/50-mod-http-geoip.conf:1
Apr 18 14:53:51 onelink5 nginx[160910]: nginx: configuration file /etc/nginx/nginx.conf test failed
Apr 18 14:53:51 onelink5 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Apr 18 14:53:51 onelink5 systemd[1]: nginx.service: Failed with result 'exit-code'.
Apr 18 14:53:51 onelink5 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
dpkg: error processing package nginx (--configure):
 installed nginx package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-filter:
 libnginx-mod-http-brotli-filter depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-filter (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          dpkg: dependency problems prevent configuration of nginx-common:
 nginx-common depends on nginx (>= 1.25.5-2+ubuntu22.04.1+deb.sury.org+1); however:
  Package nginx is not configured yet.
 nginx-common depends on nginx (<< 1.25.5-2+ubuntu22.04.1+deb.sury.org+1.1~); however:
  Package nginx is not configured yet.

dpkg: error processing package nginx-common (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Errors were encountered while processing:
 nginx
 libnginx-mod-http-brotli-filter
 nginx-common
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Expected behavior NGINX upgrade completes without issue

Distribution (please complete the following information):

Package(s) (please complete the following information):

sudo apt-cache policy nginx
nginx:
  Installed: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Candidate: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Version table:
 *** 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1.25.4-1+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
     1.18.0-6ubuntu14.4 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.18.0-6ubuntu14.3 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     1.18.0-6ubuntu14 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

sudo apt-cache policy nginx-core
nginx-core:
  Installed: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Candidate: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Version table:
 *** 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1.18.0-6ubuntu14.4 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.18.0-6ubuntu14.3 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     1.18.0-6ubuntu14 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

sudo apt-cache policy nginx-common
nginx-common:
  Installed: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Candidate: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Version table:
 *** 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1.25.4-1+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
     1.18.0-6ubuntu14.4 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.18.0-6ubuntu14.3 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     1.18.0-6ubuntu14 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Additional context Error removing the brotli package which has since been replaced by the two split packages:

sudo apt remove libnginx-mod-brotli
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libnginx-mod-http-geoip : Depends: nginx-common (= 1.25.3-1+ubuntu22.04.1+deb.sury.org+1) but 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 is to be installed
 nginx-core : Depends: libnginx-mod-http-geoip (>= 1.25.5-2+ubuntu22.04.1+deb.sury.org+1) but 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 is to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

If I attempt to use "fix broken install" it threatens to remove a bunch of nginx packages again:

sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libnginx-mod-brotli libnginx-mod-http-brotli-static libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-ssl-ct libnginx-mod-stream libnginx-mod-stream-geoip nginx-core
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libnginx-mod-brotli libnginx-mod-http-brotli-static libnginx-mod-http-geoip
The following NEW packages will be installed:
  libnginx-mod-http-brotli-static
The following packages will be upgraded:
  libnginx-mod-brotli libnginx-mod-http-geoip
2 upgraded, 1 newly installed, 0 to remove and 45 not upgraded.
11 not fully installed or removed.
Need to get 0 B/50.2 kB of archives.
After this operation, 33.8 kB disk space will be freed.
Do you want to continue? [Y/n] n

Current list of packages installed:

libnginx-mod-brotli/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6]
libnginx-mod-http-brotli-filter/jammy,now 1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6 amd64 [installed]
libnginx-mod-http-geoip/now 1.25.3-1+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,upgradable to: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1]
libnginx-mod-http-image-filter/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,auto-removable]
libnginx-mod-http-xslt-filter/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,auto-removable]
libnginx-mod-mail/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,auto-removable]
libnginx-mod-ssl-ct/jammy,now 1:1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6 amd64 [installed,auto-removable]
libnginx-mod-stream-geoip/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,auto-removable]
libnginx-mod-stream/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 amd64 [installed,auto-removable]
nginx-common/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 all [installed,automatic]
nginx-core/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 all [installed,auto-removable]
nginx/jammy,now 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 amd64 [installed]

Any help appreciated. Also please let me know if I can provide any additional debugging information. Thanks!

oerdnj commented 2 months ago

The libnginx-mod-http-geoip didn’t get upgraded and that’s preventing the upgrade to finish. Now the question is why.

oerdnj commented 2 months ago

You can try installing it manually.

ahills60 commented 2 months ago

I suspect it's related as I had a similar problem. Whilst I have libnginx-mod-http-geoip installed, my nginx service failed to restart because:

Apr 18 22:28:34 puck nginx[2916590]: 2024/04/18 22:28:34 [emerg] 2916590#2916590: module "/usr/share/nginx/modules/ngx_ssl_ct_module.so" version 1025004 ins
tead of 1025005 in /etc/nginx/modules-enabled/50-mod-ssl-ct.conf:1

Running sudo apt --fix-broken install would attempt to install and upgrade both libnginx-mod-brotli and libnginx-mod-ssl-ct and continue to result in nginx attempting to restart and fail for the above reason.

I forcibly removed libnginx-mod-brotli (i.e. sudo dpkg --remove libnginx-mod-brotli), ran sudo apt update and attempted sudo apt upgrade again. The libnginx-mod-ssl-ct package installed without issue and nginx is now running again. It would appear libnginx-mod-brotli may be the cause?

oerdnj commented 2 months ago

Run dpkg -i /var/cache/apt/archives/<package>.deb for all modules that has wrong API.

The current Debian packaging that got inherited from upstream-Debian is clearly wrong.

It needs delayed restart, it's broken even for its own modules.

ComputerTinker commented 2 months ago
sudo dpkg -i /var/cache/apt/archives/libnginx-mod-brotli_1%3a1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6_all.deb
dpkg: regarding .../libnginx-mod-brotli_1%3a1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6_all.deb containing libnginx-mod-brotli, pre-dependency problem:
 libnginx-mod-brotli pre-depends on libnginx-mod-http-brotli-filter
  libnginx-mod-http-brotli-filter is unpacked, but has never been configured.

dpkg: error processing archive /var/cache/apt/archives/libnginx-mod-brotli_1%3a1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6_all.deb (--install):
 pre-dependency problem - not installing libnginx-mod-brotli
Errors were encountered while processing:
 /var/cache/apt/archives/libnginx-mod-brotli_1%3a1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6_all.deb
sudo dpkg -i /var/cache/apt/archives/libnginx-mod-http-brotli-filter_1%3a1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6_amd64.deb
(Reading database ... 117026 files and directories currently installed.)
Preparing to unpack .../libnginx-mod-http-brotli-filter_1%3a1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6_amd64.deb ...
Unpacking libnginx-mod-http-brotli-filter (1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6) over (1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6) ...
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-filter:
 libnginx-mod-http-brotli-filter depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-filter (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 libnginx-mod-http-brotli-filter
sudo dpkg -i /var/cache/apt/archives/nginx_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb
(Reading database ... 117026 files and directories currently installed.)
Preparing to unpack .../nginx_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
dpkg: dependency problems prevent configuration of nginx:
 nginx depends on nginx-common (= 1.25.5-2+ubuntu22.04.1+deb.sury.org+1); however:
  Package nginx-common is not configured yet.

dpkg: error processing package nginx (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.10.2-1) ...
Errors were encountered while processing:
 nginx
sudo dpkg -i /var/cache/apt/archives/nginx-common_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_all.deb
(Reading database ... 117026 files and directories currently installed.)
Preparing to unpack .../nginx-common_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_all.deb ...
Unpacking nginx-common (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
dpkg: dependency problems prevent configuration of nginx-common:
 nginx-common depends on nginx (>= 1.25.5-2+ubuntu22.04.1+deb.sury.org+1); however:
  Package nginx is not configured yet.
 nginx-common depends on nginx (<< 1.25.5-2+ubuntu22.04.1+deb.sury.org+1.1~); however:
  Package nginx is not configured yet.

dpkg: error processing package nginx-common (--install):
 dependency problems - leaving unconfigured
Processing triggers for ufw (0.36.1-4ubuntu0.1) ...
Errors were encountered while processing:
 nginx-common
sudo dpkg -i /var/cache/apt/archives/nginx-common_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_all.deb /var/cache/apt/archives/nginx_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb
(Reading database ... 117026 files and directories currently installed.)
Preparing to unpack .../nginx-common_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_all.deb ...
Unpacking nginx-common (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../nginx_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Setting up nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
invoke-rc.d: initscript nginx, action "start" failed.
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2024-04-19 08:15:09 EDT; 25ms ago
       Docs: man:nginx(8)
    Process: 237315 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 8ms

Apr 19 08:15:09 onelink5 systemd[1]: Starting A high performance web server and a reverse proxy server...
Apr 19 08:15:09 onelink5 nginx[237315]: 2024/04/19 08:15:09 [emerg] 237315#237315: module "/usr/share/nginx/modules/ngx_http_geoip_module.so" version 1025003 instead of 1025005 in /etc/nginx/modules-enabled/50-mod-http-geoip.conf:1
Apr 19 08:15:09 onelink5 nginx[237315]: nginx: configuration file /etc/nginx/nginx.conf test failed
Apr 19 08:15:09 onelink5 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Apr 19 08:15:09 onelink5 systemd[1]: nginx.service: Failed with result 'exit-code'.
Apr 19 08:15:09 onelink5 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
dpkg: error processing package nginx (--install):
 installed nginx package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nginx-common:
 nginx-common depends on nginx (<< 1.25.5-2+ubuntu22.04.1+deb.sury.org+1.1~); however:
  Package nginx is not configured yet.

dpkg: error processing package nginx-common (--install):
 dependency problems - leaving unconfigured
Processing triggers for ufw (0.36.1-4ubuntu0.1) ...
Processing triggers for man-db (2.10.2-1) ...
Errors were encountered while processing:
 nginx
 nginx-common
patschi commented 2 months ago

I forcibly removed libnginx-mod-brotli (i.e. sudo dpkg --remove libnginx-mod-brotli), ran sudo apt update and attempted sudo apt upgrade again. The libnginx-mod-ssl-ct package installed without issue and nginx is now running again. It would appear libnginx-mod-brotli may be the cause?

This just saved me quite a lot frustration and downtime. Thank you so, so much, @ahills60!

This is a host where nginx just blew up after apt upgrade:

# apt install -f
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libnginx-mod-brotli libnginx-mod-http-dav-ext libnginx-mod-http-headers-more-filter
The following packages will be upgraded:
  libnginx-mod-brotli libnginx-mod-http-dav-ext libnginx-mod-http-headers-more-filter
3 upgraded, 0 newly installed, 0 to remove and 51 not upgraded.
6 not fully installed or removed.
Need to get 0 B/33.4 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
invoke-rc.d: initscript nginx, action "start" failed.
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2024-04-19 16:53:10 UTC; 28ms ago
       Docs: man:nginx(8)
    Process: 611527 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 11ms

Apr 19 16:53:10 serverFQDN systemd[1]: Starting A high performance web server and a reverse proxy server...
Apr 19 16:53:10 serverFQDN nginx[611527]: 2024/04/19 16:53:10 [emerg] 611527#611527: module "/usr/share/nginx/modules/ngx_http_dav_ext_module.so" version 1025004 instead of 1025005 in /etc/nginx/modules-enabled/50-mod-http-dav-ext.conf:1
Apr 19 16:53:10 serverFQDN nginx[611527]: nginx: configuration file /etc/nginx/nginx.conf test failed
Apr 19 16:53:10 serverFQDN systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Apr 19 16:53:10 serverFQDN systemd[1]: nginx.service: Failed with result 'exit-code'.
Apr 19 16:53:10 serverFQDN systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
dpkg: error processing package nginx (--configure):
 installed nginx package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-filter:
 libnginx-mod-http-brotli-filter depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-filter (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-static:
 libnginx-mod-http-brotli-static depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-static (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of nginx-common:
 nginx-common depends on nginx (>= 1.25.5-2+ubuntu22.04.1+deb.sury.org+1); however:
  Package nginx is not configured yet.
 nginx-common depends on nginx (<< 1.25.5-2+ubuntu22.04.1+deb.sury.org+1.1~); however:
  Package nginx is not configured yet.

dpkg: error processing package nginx-common (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 nginx
 libnginx-mod-http-brotli-filter
 libnginx-mod-http-brotli-static
 nginx-common
E: Sub-process /usr/bin/dpkg returned an error code (1)

Then running as suggested:

# dpkg --remove libnginx-mod-brotli
(Reading database ... 115816 files and directories currently installed.)
Removing libnginx-mod-brotli (1:1.0.0~rc+1.25.4-8+ubuntu22.04.1+deb.sury.org+1) ...

After removing above:

$ apt install -f
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  libnginx-mod-http-dav-ext libnginx-mod-http-headers-more-filter
The following packages will be upgraded:
  libnginx-mod-http-dav-ext libnginx-mod-http-headers-more-filter
2 upgraded, 0 newly installed, 0 to remove and 51 not upgraded.
6 not fully installed or removed.
Need to get 0 B/29.3 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
(Reading database ... 115813 files and directories currently installed.)
Preparing to unpack .../libnginx-mod-http-dav-ext_1%3a3.0.0+1.25.5-5+ubuntu22.04.1+deb.sury.org+6_amd64.deb ...
Unpacking libnginx-mod-http-dav-ext (1:3.0.0+1.25.5-5+ubuntu22.04.1+deb.sury.org+6) over (1:3.0.0+1.25.4-5+ubuntu22.04.1+deb.sury.org+4) ...
Preparing to unpack .../libnginx-mod-http-headers-more-filter_1%3a0.37+1.25.5-2+ubuntu22.04.1+deb.sury.org+6_amd64.deb ...
Unpacking libnginx-mod-http-headers-more-filter (1:0.37+1.25.5-2+ubuntu22.04.1+deb.sury.org+6) over (1:0.37+1.25.4-2+ubuntu22.04.1+deb.sury.org+4) ...
Setting up nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Setting up libnginx-mod-http-headers-more-filter (1:0.37+1.25.5-2+ubuntu22.04.1+deb.sury.org+6) ...
Setting up nginx-common (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Setting up libnginx-mod-http-brotli-filter (1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6) ...
Setting up libnginx-mod-http-echo (1:0.63+1.25.5-6+ubuntu22.04.1+deb.sury.org+6) ...
Setting up nginx-light (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Setting up libnginx-mod-http-dav-ext (1:3.0.0+1.25.5-5+ubuntu22.04.1+deb.sury.org+6) ...
Setting up libnginx-mod-http-brotli-static (1:1.0.0~rc+1.25.5-8+ubuntu22.04.1+deb.sury.org+6) ...
Processing triggers for ufw (0.36.1-4ubuntu0.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Triggering nginx reload ...

Upgrade went through with no errors and nginx is back up and running:

$ systemctl status nginx | grep -i active
     Active: active (running) since Fri 2024-04-19 16:55:18 UTC; 12s ago

This was a "fresh" broken system, with only apt upgrade run before. I did not mess around with any configurations, packets or whatsoever to try fixing it on this particular host. Just by removing above package, everything went smooth again. So it looks like it's indeed related to the brotli package.

TaylanUB commented 2 months ago

I've had a similar issue today on Debian 12 (Bookworm) using the nginx-mainline repo, and having nginx-extras installed, as well as a few modules explicitly (which I think weren't provided by extras).

I'm offering this information in case it helps someone, because the cases described above seemed different from mine. I also have a solution that doesn't involve downtime (no removing of any packages, only downgrades) so long as you haven't explicitly stopped nginx.

Symptoms

After an update, upgrade wants to do the following:

root@wiki ~ # apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  libnginx-mod-http-geoip libnginx-mod-http-geoip2 libnginx-mod-http-image-filter
  libnginx-mod-http-lua libnginx-mod-http-perl libnginx-mod-http-xslt-filter libnginx-mod-mail
  libnginx-mod-stream libnginx-mod-stream-geoip libnginx-mod-stream-geoip2 linux-image-amd64 nginx
  nginx-common nginx-extras
The following packages will be upgraded:
  libnginx-mod-brotli libnginx-mod-http-auth-pam libnginx-mod-http-brotli-filter
  libnginx-mod-http-brotli-static libnginx-mod-http-cache-purge libnginx-mod-http-dav-ext
  libnginx-mod-http-echo libnginx-mod-http-fancyindex libnginx-mod-http-headers-more-filter
  libnginx-mod-http-ndk libnginx-mod-http-subs-filter libnginx-mod-http-uploadprogress
  libnginx-mod-http-upstream-fair libnginx-mod-nchan
14 upgraded, 0 newly installed, 0 to remove and 14 not upgraded.
Need to get 0 B/409 kB of archives.
After this operation, 1,024 B disk space will be freed.
Do you want to continue? [Y/n]

So it holds back the actual nginx package (which is on 1.25.4) and a few modules, but is fine with upgrading some other modules (to 1.25.5).

Afterwards, nginx fails to reload due to the new 1.25.5 modules being incompatible with the still installed 1.25.4 version:

Apr 19 19:25:23 wiki systemd[1]: Reloading nginx.service - A high performance web server and a reverse proxy server...
Apr 19 19:25:23 wiki nginx[3497187]: 2024/04/19 19:25:23 [emerg] 3497187#3497187: module "/usr/share/nginx/modules/ndk_http_module.so" version 1025005 instead of 1025004 in /etc/nginx/modules-enabled/10-mod-http-ndk.conf:1
Apr 19 19:25:23 wiki systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Apr 19 19:25:23 wiki systemd[1]: Reload failed for nginx.service - A high performance web server and a reverse proxy server.

Solution

I've resolved this by manually reinstalling the old (1.25.4) versions of the modules that were upgraded, e.g.:

dpkg -i /var/cache/apt/archives/libnginx-mod-http-brotli-static_1%3a1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f_amd64.deb

I generally use apt-get directly instead of apt. I've heard that apt would have removed the old packages, so good thing I don't use it.

To find the exact version strings that make up the file names in /var/cache/apt/archives, you can check /var/log/dpkg.log. Note that colon characters (:) are escaped as %3a in the filenames.

Every time you do the above for a module, it will warn about the downgrade and also try to reload nginx and fail, so long as you've not reached the last downgrade yet:

root@wiki ~ # dpkg -i /var/cache/apt/archives/libnginx-mod-http-brotli-static_1%3a1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f_amd64.deb
dpkg: warning: downgrading libnginx-mod-http-brotli-static from 1:1.0.0~rc+1.25.5-8+0~20240418.14+debian12~1.gbpf34a5f to 1:1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f
(Reading database ... 60649 files and directories currently installed.)
Preparing to unpack .../libnginx-mod-http-brotli-static_1%3a1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f_amd64.deb ...
Unpacking libnginx-mod-http-brotli-static (1:1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f) over (1:1.0.0~rc+1.25.5-8+0~20240418.14+debian12~1.gbpf34a5f) ...
Setting up libnginx-mod-http-brotli-static (1:1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f) ...
Processing triggers for nginx (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b) ...
Triggering nginx reload ...
Job for nginx.service failed.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
invoke-rc.d: initscript nginx, action "reload" failed.
root@wiki ~ #

That's OK and you can ignore it. When you reach the last module that needs the downgrade, the nginx reload should work, since no broken modules will be left:

root@wiki ~ # dpkg -i /var/cache/apt/archives/libnginx-mod-nchan_1%3a1.3.6+dfsg+1.25.4-4+0~20240226.1+debian12~1.gbpda3c7f_amd64.deb
dpkg: warning: downgrading libnginx-mod-nchan from 1:1.3.6+dfsg+1.25.5-4+0~20240418.3+debian12~1.gbpda3c7f to 1:1.3.6+dfsg+1.25.4-4+0~20240226.1+debian12~1.gbpda3c7f
(Reading database ... 60649 files and directories currently installed.)
Preparing to unpack .../libnginx-mod-nchan_1%3a1.3.6+dfsg+1.25.4-4+0~20240226.1+debian12~1.gbpda3c7f_amd64.deb ...
Unpacking libnginx-mod-nchan (1:1.3.6+dfsg+1.25.4-4+0~20240226.1+debian12~1.gbpda3c7f) over (1:1.3.6+dfsg+1.25.5-4+0~20240418.3+debian12~1.gbpda3c7f) ...
Setting up libnginx-mod-nchan (1:1.3.6+dfsg+1.25.4-4+0~20240226.1+debian12~1.gbpda3c7f) ...
Processing triggers for nginx (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b) ...
Triggering nginx reload ...
root@wiki ~ #

Thankfully I had no downtime at all through the entire ordeal, since nginx just kept running despite all the failed reloads. Phew!

oerdnj commented 2 months ago

So, this is absolutely wrong:

The following packages have been kept back:
  libnginx-mod-http-geoip libnginx-mod-http-geoip2 libnginx-mod-http-image-filter
  libnginx-mod-http-lua libnginx-mod-http-perl libnginx-mod-http-xslt-filter libnginx-mod-mail
  libnginx-mod-stream libnginx-mod-stream-geoip libnginx-mod-stream-geoip2 linux-image-amd64 nginx
  nginx-common nginx-extras

I would like to know why this has happened?

What happens if you do apt-get install libnginx-mod-stream-geoip nginx?

TaylanUB commented 2 months ago

Here's the output of that (with -V for more clarity) after another apt-get update just now:

root@wiki ~ # apt-get install -V libnginx-mod-stream-geoip nginx
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
   libluajit-5.1-2 (2.1.0+openresty20231117-2+0~20240418.7+debian12~1.gbp0a862c)
   libluajit-5.1-common (2.1.0+openresty20231117-2+0~20240418.7+debian12~1.gbp0a862c)
   libnginx-mod-http-geoip (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-image-filter (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-perl (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-xslt-filter (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-mail (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   lua-resty-lrucache (0.13-10+0~20240226.1+debian12~1.gbp686f29)
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
   libnginx-mod-brotli (1:1.0.0~rc+1.25.5-9+0~20240420.15+debian12~1.gbp35d06f)
   libnginx-mod-http-brotli-filter (1:1.0.0~rc+1.25.5-9+0~20240420.15+debian12~1.gbp35d06f)
   libnginx-mod-http-brotli-static (1:1.0.0~rc+1.25.5-9+0~20240420.15+debian12~1.gbp35d06f)
   libnginx-mod-http-geoip (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-image-filter (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-perl (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-xslt-filter (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-mail (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-stream (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   nginx-common (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
Suggested packages:
   fcgiwrap (1.1.0-14)
   nginx-doc (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
The following packages will be REMOVED:
   libnginx-mod-http-auth-pam (1:1.5.5+1.25.5-2+0~20240418.4+debian12~1.gbpc07838)
   libnginx-mod-http-cache-purge (1:2.3+1.25.5-7~20240418.5+debian12)
   libnginx-mod-http-dav-ext (1:3.0.0+1.25.5-5+0~20240418.4+debian12~1.gbp2ebee6)
   libnginx-mod-http-echo (1:0.63+1.25.5-6+0~20240418.3+debian12~1.gbp641470)
   libnginx-mod-http-fancyindex (1:0.5.2+1.25.5-5+0~20240418.3+debian12~1.gbpc55420)
   libnginx-mod-http-geoip2 (1:3.4+1.25.4-5+0~20240226.1+debian12~1.gbpf3955f)
   libnginx-mod-http-headers-more-filter (1:0.37+1.25.5-2+0~20240418.3+debian12~1.gbp554baf)
   libnginx-mod-http-lua (1:0.10.26+1.25.5-2+0~20240418.11+debian12~1.gbpb6cd61)
   libnginx-mod-http-ndk (1:0.3.3+1.25.5-1+0~20240418.3+debian12~1.gbp4eef54)
   libnginx-mod-http-subs-filter (1:0.6.4+1.25.5-6+0~20240418.3+debian12~1.gbpd1bc1d)
   libnginx-mod-http-uploadprogress (1:0.9.2+1.25.5-5+0~20240418.3+debian12~1.gbp741e9d)
   libnginx-mod-http-upstream-fair (1:0.0~git20120408.a18b409+1.25.5-5+0~20240418.5+debian12~1.gbp76b535)
   libnginx-mod-nchan (1:1.3.6+dfsg+1.25.5-4+0~20240418.3+debian12~1.gbpda3c7f)
   libnginx-mod-stream-geoip2 (1:3.4+1.25.4-5+0~20240226.1+debian12~1.gbpf3955f)
   lua-resty-core (0.1.28-2+0~20240226.2+debian12~1.gbp0e1ea3)
   nginx-extras (1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
The following packages will be upgraded:
   libnginx-mod-brotli (1:1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f => 1:1.0.0~rc+1.25.5-9+0~20240420.15+debian12~1.gbp35d06f)
   libnginx-mod-http-brotli-filter (1:1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f => 1:1.0.0~rc+1.25.5-9+0~20240420.15+debian12~1.gbp35d06f)
   libnginx-mod-http-brotli-static (1:1.0.0~rc+1.25.4-8+0~20240321.10+debian12~1.gbpf34a5f => 1:1.0.0~rc+1.25.5-9+0~20240420.15+debian12~1.gbp35d06f)
   libnginx-mod-http-geoip (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-image-filter (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-perl (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-http-xslt-filter (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-mail (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-stream (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   libnginx-mod-stream-geoip (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   nginx (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
   nginx-common (1.25.4-4+0~20240320.50+debian12~1.gbp381c4b => 1.25.5-2+0~20240419.55+debian12~1.gbp2f358e)
12 upgraded, 0 newly installed, 16 to remove and 1 not upgraded.
Need to get 1,467 kB of archives.
After this operation, 2,143 kB disk space will be freed.
Do you want to continue? [Y/n]
TaylanUB commented 2 months ago

The Depends: line for the modules it wants to remove seem suspicious. They depend on nginx-abi-1.24.0-1.

Exhibit A:

root@wiki ~ # apt-cache show libnginx-mod-http-auth-pam
[...]
Version: 1:1.5.5+1.25.5-2+0~20240418.4+debian12~1.gbpc07838
[...]
Depends: nginx-abi-1.24.0-1, libc6 (>= 2.14), libpam0g (>= 0.99.7.1)
[...]

Exhibit B:

root@wiki ~ # apt-cache show libnginx-mod-http-cache-purge
[...]
Version: 1:2.3+1.25.5-7~20240418.5+debian12
[...]
Depends: nginx-abi-1.24.0-1, libc6 (>= 2.14)
[...]

The ones it wants to upgrade instead depend on nginx-abi-1.25.5-1. So it seems the issue may be that some of the modules had their version bumped to 1.25.5 but their nginx-abi-... dependency wasn't updated?

oerdnj commented 2 months ago

I've issue new rebuild now and I see:

$ apt-cache show libnginx-mod-http-cache-purge
Package: libnginx-mod-http-cache-purge
Source: libnginx-mod-http-cache-purge (1:2.3-7~20240420.6+debian12)
Version: 1:2.3+1.25.5-7~20240420.6+debian12
Architecture: amd64
Maintainer: Debian Nginx Maintainers <pkg-nginx-maintainers@alioth-lists.debian.net>
Installed-Size: 42
Depends: nginx-abi-1.25.5-1, libc6 (>= 2.14)
Recommends: nginx
Multi-Arch: foreign
Homepage: https://github.com/FRiCKLE/ngx_cache_purge
Priority: optional
Section: httpd
Filename: pool/main/libn/libnginx-mod-http-cache-purge/libnginx-mod-http-cache-purge_2.3+1.25.5-7~20240420.6+debian12_amd64.deb
Size: 10584
SHA256: f68188538314ff39e14dc87115781707a8cc149c34df8839b198e1bc53fe469b
SHA1: 4f261b1af4477c0830f3ef01c75688517ebb4be4
MD5sum: d048fa0d6b1194c832a57b3d3dc81355
Description-en: Purge content from Nginx caches
 Cache Purge module adds purging capabilities to Nginx. It allows purging
 content from caches used by all of Nginx proxy modules, like FastCGI, Proxy,
 SCGI and uWSGI.
Description-md5: 398ab43d3187cbe8c5005bbdf7b75662

and

Package: libnginx-mod-http-auth-pam
Source: libnginx-mod-http-auth-pam (1:1.5.5-2+0~20240420.5+debian12~1.gbpc07838)
Version: 1:1.5.5+1.25.5-2+0~20240420.5+debian12~1.gbpc07838
Architecture: amd64
Maintainer: Debian Nginx Maintainers <pkg-nginx-maintainers@alioth-lists.debian.net>
Installed-Size: 35
Depends: nginx-abi-1.25.5-1, libc6 (>= 2.14), libpam0g (>= 0.99.7.1)
Recommends: nginx
Multi-Arch: foreign
Homepage: https://github.com/sto/ngx_http_auth_pam_module
Priority: optional
Section: httpd
Filename: pool/main/libn/libnginx-mod-http-auth-pam/libnginx-mod-http-auth-pam_1.5.5+1.25.5-2+0~20240420.5+debian12~1.gbpc07838_amd64.deb
Size: 8632
SHA256: a792baab107c54a40af8f7e34da6be2e47b2a954d85b3eeb1a819d2bb2a965af
SHA1: 7e9ffbb32fc13b5ea3646b4538b66c98c144b109
MD5sum: d419f00a41ef8121720566e251fdcfad
Description-en: PAM authentication module for Nginx
 The nginx_http_auth_pam module enables authentication using PAM.
 .
 The module uses PAM as a backend for simple http authentication. It
 also allows setting the pam service name to allow more fine grained control.
Description-md5: f5d57dbf0c65acd242b1a0560e3e3bfd
TaylanUB commented 2 months ago

That seems to have fixed the issue for those two. After another apt-get update it now wants to upgrade rather than remove them when I run apt-get install nginx (i.e. try to upgrade nginx to 1.25.5).

The following still seem to have the issue, depending on nginx-abi-1.24.0-1:

libnginx-mod-http-dav-ext
libnginx-mod-http-echo
libnginx-mod-http-fancyindex
libnginx-mod-http-geoip2
libnginx-mod-http-headers-more-filter
libnginx-mod-http-lua
libnginx-mod-http-ndk
libnginx-mod-http-subs-filter
libnginx-mod-http-uploadprogress
libnginx-mod-http-upstream-fair
libnginx-mod-nchan
libnginx-mod-stream-geoip2

(The packages lua-resty-core and nginx-extras only seem to suffer from the problem indirectly, due to depending on one of those listed above.)

oerdnj commented 2 months ago

Looks like something also failed with Debian nginx rebuild.

Check for the ABI dependencies:

# cat /tmp/p
libnginx-mod-http-auth-pam
libnginx-mod-http-brotli-filter
libnginx-mod-http-brotli-static
libnginx-mod-brotli
libnginx-mod-http-cache-purge
libnginx-mod-http-dav-ext
libnginx-mod-http-echo
libnginx-mod-http-fancyindex
libnginx-mod-http-geoip2
libnginx-mod-stream-geoip2
libnginx-mod-http-headers-more-filter
libnginx-mod-http-lua
libnginx-mod-http-ndk
libnginx-mod-http-ndk-dev
libnginx-mod-http-subs-filter
libnginx-mod-http-uploadprogress
libnginx-mod-http-upstream-fair
libnginx-mod-nchan
libnginx-mod-rtmp
libnginx-mod-ssl-ct
sudo apt update && cat /tmp/p | while read P; do apt-cache show $P | grep -E "^(Package|Version|Depends)" | head -3; done | sed -e 's/Depends: .*, nginx-abi/Depends: nginx-abi/;s/,.*//' | grep 1.24.0 -B 2 | grep ^Package

On Ubuntu Focal and Jammy, this produces empty list;

On Debian bookworm, this produced:

Package: libnginx-mod-http-dav-ext
Package: libnginx-mod-http-echo
Package: libnginx-mod-http-fancyindex
Package: libnginx-mod-http-geoip2
Package: libnginx-mod-stream-geoip2
Package: libnginx-mod-http-headers-more-filter
Package: libnginx-mod-http-lua
Package: libnginx-mod-http-ndk
Package: libnginx-mod-http-subs-filter
Package: libnginx-mod-http-uploadprogress
Package: libnginx-mod-http-upstream-fair
Package: libnginx-mod-nchan
Package: libnginx-mod-rtmp
Package: libnginx-mod-ssl-ct

I've issued yet another rebuild for all of these.

Thank you, @TaylanUB, for pointing me into the right direction!!!

TaylanUB commented 2 months ago

Thank you for taking care of it so swiftly! :-)

The following three still seem to have the issue. Don't know if they're still rebuilding or what.

   libnginx-mod-http-geoip2 (1:3.4+1.25.5-5+0~20240418.3+debian12~1.gbpf3955f)
   libnginx-mod-http-lua (1:0.10.26+1.25.5-2+0~20240418.11+debian12~1.gbpb6cd61)
   libnginx-mod-stream-geoip2 (1:3.4+1.25.5-5+0~20240418.3+debian12~1.gbpf3955f)
oerdnj commented 2 months ago

Those need the first round to finish building first because there’s a dependency on other modules to build first.

oerdnj commented 2 months ago

And no, thank you, @TaylanUB, I was kind of desperate as the new packaging that I had to adopt kind of break all the hell loose.

szhu25 commented 2 months ago

I was able to follow this issue and manually fix my broken Nginx (and I did not realize it's in real time!). Thanks!

oerdnj commented 2 months ago

This now looks good on my side:

ondrej@calcifer:~$ cat /tmp/p | while read P; do apt-cache show $P | grep -E "^(Package|Version|Depends)" | head -3; done | sed -e 's/Depends: .*, nginx-abi/Depends: nginx-abi/;s/,.*//' | grep 1.24.0 -B 2 | grep ^Package
ondrej@calcifer:~$
oerdnj commented 2 months ago

I think this should not be all rebuilt and have correct dependencies on the 1.25.5 ABI. Please open new issue if you still have a problem, a pretty please provide complete information - I especially need to know results of apt-cache show <failing_package> and apt-cache policy <failing_package>. Please provide as much information as you can (properly formatted) as there could be a bit of information I might be missing included in the output. See here for an example of "the bit" that helped: https://github.com/oerdnj/deb.sury.org/issues/2119#issuecomment-2067689407

fliespl commented 1 month ago

@oerdnj not sure if connected, but I have been getting same issues on mainline, while it worked with normal repository.

May 01 20:00:44 ip-172-26-9-103 nginx[25798]: 2024/05/01 20:00:44 [emerg] 25798#25798: module "/usr/share/nginx/modules/ngx_ssl_ct_module.so" version 1025004 instead of 1025005 in /etc/nginx/modules-enabled/50-mod-ssl-ct.conf:1
May 01 20:00:44 ip-172-26-9-103 nginx[25798]: nginx: configuration file /etc/nginx/nginx.conf test failed
May 01 20:00:44 ip-172-26-9-103 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
May 01 20:00:44 ip-172-26-9-103 systemd[1]: nginx.service: Failed with result 'exit-code'.
May 01 20:00:44 ip-172-26-9-103 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
dpkg: error processing package nginx (--configure):
 installed nginx package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-filter:
 libnginx-mod-http-brotli-filter depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-filter (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-static:
 libnginx-mod-http-brotli-static depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-static (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of nginx-common:
 nginx-common depends on nginx (>= 1.25.5-2+0~20240419.55+debian10~1.gbp2f358e); however:
  Package nginx is not configured yet.
 nginx-common depends on nginx (<< 1.25.5-2+0~20240419.55+debian10~1.gbp2f358e.1~); however:
  Package nginx is not configured yet.

dpkg: error processing package nginx-common (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 nginx
 libnginx-mod-http-brotli-filter
 libnginx-mod-http-brotli-static
 nginx-common
E: Sub-process /usr/bin/dpkg returned an error code (1)
oerdnj commented 1 month ago

@fliespl Excellent (not for you, but for the debugging, sorry...)

Can you do apt-cache policy libnginx-mod-ssl-ct and paste output here?

Mine looks like this:

Package: libnginx-mod-ssl-ct
Source: libnginx-mod-ssl-ct (1:1.3.2-2+0~20240420.6+debian12~1.gbpe4bbad)
Version: 1:1.3.2+1.25.5-2+0~20240420.6+debian12~1.gbpe4bbad
Architecture: amd64
Maintainer: Debian Nginx Maintainers <pkg-nginx-maintainers@alioth-lists.debian.net>
Installed-Size: 48
Depends: nginx-abi-1.25.5-1, libc6 (>= 2.33)
Recommends: nginx
Multi-Arch: foreign
Homepage: https://github.com/grahamedgecombe/nginx-ct
Priority: optional
Section: httpd
Filename: pool/main/libn/libnginx-mod-ssl-ct/libnginx-mod-ssl-ct_1.3.2+1.25.5-2+0~20240420.6+debian12~1.gbpe4bbad_amd64.deb
Size: 7520
SHA256: 99170011b627f022c829f39bd23e93c80b647a69655ccaec6efded7dedc4b09d
SHA1: 6374333570b1476e8ecc344a7a69fed37b0ff009
MD5sum: 3647c8cf93d24090d997c06f83776143
Description: adds support for the TLS signed_certificate_timestamp extension
 This module adds support for the TLS signed_certificate_timestamp extension to
 nginx, which is one of the mechanisms supported by Google's Certificate
 Transparency project to deliver Signed Certificate Timestamps to TLS clients.
Description-md5: b2ab35d0290701c6baffc537e23d0834

Package: libnginx-mod-ssl-ct
Source: libnginx-mod-ssl-ct (1:1.3.2-2+0~20240316.2+debian12~1.gbpe4bbad)
Version: 1:1.3.2+1.24.0-2+0~20240316.2+debian12~1.gbpe4bbad
Architecture: amd64
Maintainer: Debian Nginx Maintainers <pkg-nginx-maintainers@alioth-lists.debian.net>
Installed-Size: 48
Depends: nginx-abi-1.24.0-1, libc6 (>= 2.33)
Recommends: nginx
Multi-Arch: foreign
Homepage: https://github.com/grahamedgecombe/nginx-ct
Priority: optional
Section: httpd
Filename: pool/main/libn/libnginx-mod-ssl-ct/libnginx-mod-ssl-ct_1.3.2+1.24.0-2+0~20240316.2+debian12~1.gbpe4bbad_amd64.deb
Size: 7524
SHA256: fab6f515675e8c973d49164a6e0aa1dad54f9682cdb96996fbcbee93a5d7e0e5
SHA1: d4ba86c89126e95e1d4e2a1b86b329ee8590d1de
MD5sum: 96f9e5fbe7526c20540b84d1a09a8bcf
Description: adds support for the TLS signed_certificate_timestamp extension
 This module adds support for the TLS signed_certificate_timestamp extension to
 nginx, which is one of the mechanisms supported by Google's Certificate
 Transparency project to deliver Signed Certificate Timestamps to TLS clients.
Description-md5: b2ab35d0290701c6baffc537e23d0834
fliespl commented 1 month ago

@oerdnj ouch, didn't thought you mind need more info :/ I got this while doing server OS upgrades and since it worked with normal branch I went ahead and updated everything, so setup is totally different now. Sorry.

Thomas2500 commented 1 month ago

I just wanted to update and also ran into an error:

Preconfiguring packages ...
(Reading database ... 136382 files and directories currently installed.)
Preparing to unpack .../nginx-extras_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking nginx-extras (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.4-4+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../libnginx-mod-http-brotli-filter_1%3a1.0.0~rc+1.25.5-9+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libnginx-mod-http-brotli-filter (1:1.0.0~rc+1.25.5-9+ubuntu22.04.1+deb.sury.org+1) over (1:1.0.0~rc+1.25.4-8+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../libnginx-mod-http-brotli-static_1%3a1.0.0~rc+1.25.5-9+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libnginx-mod-http-brotli-static (1:1.0.0~rc+1.25.5-9+ubuntu22.04.1+deb.sury.org+1) over (1:1.0.0~rc+1.25.4-8+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../nginx_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_amd64.deb ...
Unpacking nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.4-4+ubuntu22.04.1+deb.sury.org+1) ...
Preparing to unpack .../nginx-common_1.25.5-2+ubuntu22.04.1+deb.sury.org+1_all.deb ...
Unpacking nginx-common (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) over (1.25.4-4+ubuntu22.04.1+deb.sury.org+1) ...
Setting up nginx (1.25.5-2+ubuntu22.04.1+deb.sury.org+1) ...
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
 * Upgrading binary nginx                                                                                                                                                        [fail] 
invoke-rc.d: initscript nginx, action "upgrade" failed.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
invoke-rc.d: initscript nginx, action "restart" failed.
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-05-02 16:19:44 UTC; 40ms ago
       Docs: man:nginx(8)
    Process: 851812 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 12ms

May 02 16:19:44 multicontrol systemd[1]: Starting A high performance web server and a reverse proxy server...
May 02 16:19:44 multicontrol nginx[851812]: 2024/05/02 16:19:44 [emerg] 851812#851812: module "/usr/share/nginx/modules/ndk_http_module.so" version 1025004 instead of 1025005 in /etc/n
ginx/modules-enabled/10-mod-http-ndk.conf:1
May 02 16:19:44 multicontrol nginx[851812]: nginx: configuration file /etc/nginx/nginx.conf test failed
May 02 16:19:44 multicontrol systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
May 02 16:19:44 multicontrol systemd[1]: nginx.service: Failed with result 'exit-code'.
May 02 16:19:44 multicontrol systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
dpkg: error processing package nginx (--configure):
 installed nginx package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-filter:
 libnginx-mod-http-brotli-filter depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-filter (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-brotli-static:
 libnginx-mod-http-brotli-static depends on nginx-abi-1.25.5-1; however:
  Package nginx-abi-1.25.5-1 is not installed.
  Package nginx which provides nginx-abi-1.25.5-1 is not configured yet.

dpkg: error processing package libnginx-mod-http-brotli-static (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of nginx-common:
 nginx-common depends on nginx (>= 1.25.5-2+ubuntu22.04.1+deb.sury.org+1); however:
  Package nginx is No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                                             No apport report written because the error message indicate
s its a followup error from a previous failure.
                                               No apport report written because MaxReports is reached already
                                                                                                             not configured yet.
 nginx-common depends on nginx (<< 1.25.5-2+ubuntu22.04.1+deb.sury.org+1.1~); however:
  Package nginx is not configured yet.

dpkg: error processing package nginx-common (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 nginx
 libnginx-mod-http-brotli-filter
 libnginx-mod-http-brotli-static
 nginx-common
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Output of apt-cache policy libnginx-mod-ssl-ct:

root@multicontrol.bella.pm ~ $ apt-cache policy libnginx-mod-ssl-ct
libnginx-mod-ssl-ct:
  Installed: 1:1.3.2+1.25.4-2+ubuntu22.04.1+deb.sury.org+1
  Candidate: 1:1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6
  Version table:
     1:1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
 *** 1:1.3.2+1.25.4-2+ubuntu22.04.1+deb.sury.org+1 100
        100 /var/lib/dpkg/status

Policy of other packages:

root@multicontrol.bella.pm ~ $ apt policy nginx
nginx:
  Installed: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Candidate: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Version table:
 *** 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1.25.4-1+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
     1.18.0-6ubuntu14.4 500
        500 http://at.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.18.0-6ubuntu14.3 500
        500 http://at.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     1.18.0-6ubuntu14 500
        500 http://at.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@multicontrol.bella.pm ~ $ apt-cache policy nginx-common
nginx-common:
  Installed: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Candidate: 1.25.5-2+ubuntu22.04.1+deb.sury.org+1
  Version table:
 *** 1.25.5-2+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1.25.4-1+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
     1.18.0-6ubuntu14.4 500
        500 http://at.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.18.0-6ubuntu14.3 500
        500 http://at.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     1.18.0-6ubuntu14 500
        500 http://at.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@multicontrol.bella.pm ~ $ apt-cache policy libnginx-mod-http-brotli-filter
libnginx-mod-http-brotli-filter:
  Installed: 1:1.0.0~rc+1.25.5-9+ubuntu22.04.1+deb.sury.org+1
  Candidate: 1:1.0.0~rc+1.25.5-9+ubuntu22.04.1+deb.sury.org+1
  Version table:
 *** 1:1.0.0~rc+1.25.5-9+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

After commenting out the libs in /etc/nginx/modules-enabled/50-mod-ssl-ct.conf I retried to reapply all updates with apt --fix-broken install. This fixed the problem for me and nginx is working again.

oerdnj commented 1 month ago

@Thomas2500 Can you also do apt-cache show libnginx-mod-ssl-ct? I am trying to figure out why that package isn't upgrading with the rest and can't figure it out.

Thomas2500 commented 1 month ago

@oerdnj result is the following:

root@multicontrol.bella.pm ~ $ apt-cache show libnginx-mod-ssl-ct
Package: libnginx-mod-ssl-ct
Source: libnginx-mod-ssl-ct (1:1.3.2-2+ubuntu22.04.1+deb.sury.org+6)
Priority: optional
Section: httpd
Installed-Size: 48
Maintainer: Debian Nginx Maintainers <pkg-nginx-maintainers@alioth-lists.debian.net>
Architecture: amd64
Version: 1:1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6
Recommends: nginx
Depends: nginx-abi-1.25.5-1, libc6 (>= 2.33)
Filename: pool/main/libn/libnginx-mod-ssl-ct/libnginx-mod-ssl-ct_1.3.2+1.25.5-2+ubuntu22.04.1+deb.sury.org+6_amd64.deb
Size: 7844
MD5sum: 0af73ee436a3a2c931bf688f7a65325c
SHA1: 189f574878f730542ab735e78762f517081401d9
SHA256: 1469dc175b21f31f0d59b0233b5e5d0ca28e39b69f83016b33131c12d4ce41d9
Description-en: adds support for the TLS signed_certificate_timestamp extension
 This module adds support for the TLS signed_certificate_timestamp extension to
 nginx, which is one of the mechanisms supported by Google's Certificate
 Transparency project to deliver Signed Certificate Timestamps to TLS clients.
Description-md5: b2ab35d0290701c6baffc537e23d0834
Multi-Arch: foreign