Cockpit service will not start on Raspi zero W due to gnutls crash

[jack60612]

Explain what happens

1. connect to socket at https://IP:9090
2. socket starts cockpit service
3. service fails systemd[1]: cockpit.service: Control process exited, code=killed, status=11/SEGV
4. nothing works :(

Version of Cockpit

300.1-1~bpo12+1 (Debian

Where is the problem in Cockpit?

Applications

Server operating system

None

Server operating system version

Debian bookworm

What browsers are you using?

Chrome

System log

systemd Service:
× cockpit.service - Cockpit Web Service
     Loaded: loaded (/lib/systemd/system/cockpit.service; static)
     Active: failed (Result: signal) since Sun 2023-09-17 14:14:40 EDT; 16min ago
TriggeredBy: ● cockpit.socket
       Docs: man:cockpit-ws(8)
    Process: 1639 ExecStartPre=/usr/lib/cockpit/cockpit-certificate-ensure --for-cockpit-tls (code=killed, signal=SEGV)
        CPU: 98ms

Sep 17 14:14:40 networkpi-school systemd[1]: Starting cockpit.service - Cockpit Web Service...
Sep 17 14:14:40 networkpi-school systemd[1]: cockpit.service: Control process exited, code=killed, status=11/SEGV
Sep 17 14:14:40 networkpi-school systemd[1]: cockpit.service: Failed with result 'signal'.
Sep 17 14:14:40 networkpi-school systemd[1]: Failed to start cockpit.service - Cockpit Web Service.

[jack60612]

I also did this:

I ran strace (sudo strace /usr/lib/cockpit/cockpit-certificate-ensure --for-cockpit-tls) and it seems as if it expects a file at /etc/gnutls/config that isn't there:

statx(AT_FDCWD, "/etc/gnutls/config", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xbea740f0) = -1 ENOENT (No such file or directory)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x52c722} ---
+++ killed by SIGSEGV +++
Segmentation fault

gdb says this too:

Program received signal SIGSEGV, Segmentation fault.
0x00450722 in ?? ()

[jack60612]

it seems like the gnutls build library is out of date or something

[martinpitt]

Right, I'm afraid GnuTLS crashing is outside of our purview. Please also try GnuTLS by itself, like gnutls-cli google.com:443

[jack60612]

@martinpitt gnutls works, its a fresh install.

[jack60612]

Processed 140 CA certificate(s).
Resolving 'google.com:443'...
Connecting to '142.250.217.206:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `CN=*.google.com', issuer `CN=GTS CA 1C3,O=Google Trust Services LLC,C=US', serial 0x009bdbaae7a2280a92097ed09691a3879e, EC/ECDSA key 256 bits, signed using RSA-SHA256, activated `2023-09-04 08:17:06 UTC', expires `2023-11-27 08:17:05 UTC', pin-sha256="KIb/p/LHJD9lB1o41IczmidDvER0Ry2cvb9tRLxxfFM="
        Public Key ID:
                sha1:3fd206400bb7f4444c655af091d6225c5a3002bc
                sha256:2886ffa7f2c7243f65075a38d487339a2743bc4474472d9cbdbf6d44bc717c53
        Public Key PIN:
                pin-sha256:KIb/p/LHJD9lB1o41IczmidDvER0Ry2cvb9tRLxxfFM=

- Certificate[1] info:
 - subject `CN=GTS CA 1C3,O=Google Trust Services LLC,C=US', issuer `CN=GTS Root R1,O=Google Trust Services LLC,C=US', serial 0x0203bc53596b34c718f5015066, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-08-13 00:00:42 UTC', expires `2027-09-30 00:00:42 UTC', pin-sha256="zCTnfLwLKbS9S2sbp+uFz4KZOocFvXxkV06Ce9O5M2w="
- Certificate[2] info:
 - subject `CN=GTS Root R1,O=Google Trust Services LLC,C=US', issuer `CN=GlobalSign Root CA,OU=Root CA,O=GlobalSign nv-sa,C=BE', serial 0x77bd0d6cdb36f91aea210fc4f058d30d, RSA key 4096 bits, signed using RSA-SHA256, activated `2020-06-19 00:00:42 UTC', expires `2028-01-28 00:00:42 UTC', pin-sha256="hxqRlPTu1bMS/0DITB1SSu0vd4u/8l8TjPgfaAp63Gc="
- Status: The certificate is trusted.
- Description: (TLS1.3-X.509)-(ECDHE-X25519)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
- Session ID: 84:B4:F9:DA:46:00:24:86:49:F4:F3:2B:92:87:E6:79:4D:89:4F:D3:B1:79:5E:0E:34:36:1A:2F:55:1B:35:DE
- Options:
- Handshake was completed

- Simple Client Mode:

[martinpitt]

Can you please try gdb --args /usr/lib/cockpit/cockpit-certificate-ensure --for-cockpit-tls and get a stack trace? This probably needs to add a few debug symbol packages: https://wiki.debian.org/HowToGetABacktrace#Installing_the_debugging_symbols

[jack60612]

I'm going to get to this tomorrow, thanks for re opening it!

[jack60612]

didn't seem to give more info, so i'm downloading the source for that version '301-1~bpo12+1' (bookworm-backports) and giving it to gdb too.

[jack60612]

its not really giving me any more information.

GNU gdb (Raspbian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/cockpit/cockpit-certificate-ensure...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.debian.net>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Reading symbols from /home/jack/.cache/debuginfod_client/7917b41b21937a800daac6a7fd3fcdd566551d88/debuginfo...
(gdb) start
Temporary breakpoint 1 at 0x10b0: file src/tls/cockpit-certificate-ensure.c, line 399.
Starting program: /usr/lib/cockpit/cockpit-certificate-ensure --check
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00450722 in ?? ()
(gdb) st
Ambiguous command "st": stack, start, starti, status, step, stepi, stepping, stop, strace.
(gdb) strace
warning: Couldn't determine the static tracepoint marker to probe
Static tracepoint 2 at 0x450722
(gdb) bt
#0  0x00450722 in ?? ()
#1  0x0040156e in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

[jack60612]

@martinpitt what do you suggest?

[martinpitt]

Hmm, if the stack is that corrupted, the next step would be to build cockpit.git from source with debug info, something like:

sudo apt build-dep cockpit
./autogen.sh --prefix=/usr --with-cockpit-user=cockpit-ws --with-cockpit-ws-instance-user=cockpit-wsinstance --sysconfdir=/etc --localstatedir=/var --enable-strict --enable-debug
make -j$(nproc)
gdb --args ./cockpit-certificate-ensure --check

[jelly]

Does Raspbian lack debuginfod support? https://wiki.debian.org/Debuginfod

[jack60612]

Does Raspbian lack debuginfod support? https://wiki.debian.org/Debuginfod

From my console output above: @jelly

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.debian.net>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.

[jelly]

Does Raspbian lack debuginfod support? https://wiki.debian.org/Debuginfod

From my console output above: @jelly

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.debian.net>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.

Do you get a better backtrace now?

[jack60612]

Does Raspbian lack debuginfod support? https://wiki.debian.org/Debuginfod

From my console output above: @jelly

This GDB supports auto-downloading debuginfo from the following URLs:

  <https://debuginfod.debian.net>

Enable debuginfod for this session? (y or [n]) y

Debuginfod has been enabled.

Do you get a better backtrace now?

What I was trying to say was that I was using debuginfod already, thanks for your suggestion though.

[martinpitt]

The early stack corruption doesn't make it likely that debug symbols help, hence my suggestion to do a debug build (which does -O0)

[jack60612]

i will get to this, kinda forgot.

[jack60612]

Discovered something else interesting, when I put the same sd card (pi zero w ) into a pi 4 (2gb) it just works, so maybe its something with the CPU architectures?

[jack60612]

i still need to build, sorry for the delays

[ps1noob]

Today i ran into the exact same problem with the exact same hardware. I took the Raspberry Pi OS (bookworm) from 2023-12-11, flashed it to my SD card, booted my Pi Zero with it, included (as stated by the cockpit manual) the Debian backports, installed cockpit and.... yes, the service fails with the exact same error from @jack60612.

Cockpit is kind of an essential software for my linux machines, so is there something that can be done to get it to work in this constellation?

[martinpitt]

@ps1noob Nothing known -- this is a GnuTLS crash, someone with an affected system has to do a debug build and get a backtrace.

[ps1noob]

Maybe i could do this, but as i am more of a windows sysadmin, i'm not used to build something on a linux machine.

So, i tried your steps from above, which led me to this:

What am i missing? I thought, the apt command would download the autogen.sh to my current /tmp folder, but.... there is none:

I really would like to help out and deliver the necessary information, but without any further instruction i'm kind of stuck...

[martinpitt]

No, "apt-get build dep" installs cockpit's build dependency packages. You still need to get the actual cockpit source code to build anything. Sorry, I was missing that step from above, as developers we don't even think about it any more. As user (please don't do this as root):

apt source cockpit
cd cockpit-*

and the continue with building.

./autogen.sh --prefix=/usr --with-cockpit-user=cockpit-ws --with-cockpit-ws-instance-user=cockpit-wsinstance --sysconfdir=/etc --localstatedir=/var --enable-strict --enable-debug
make -j$(nproc)
gdb --args ./cockpit-certificate-ensure --check

[ps1noob]

Thank you for the explanation!

Well, i'm in a dead end again :(

Maybe i have to process the git command that's been told in the screenshot beforehand?

[ps1noob]

@martinpitt could you tell me what i should do next?

[martinpitt]

@ps1noob : Ah, sorry.

./configure --prefix=/usr --with-cockpit-user=cockpit-ws --with-cockpit-ws-instance-user=cockpit-wsinstance --sysconfdir=/etc --localstatedir=/var --enable-strict --enable-debug
make -j$(nproc)
gdb --args ./cockpit-certificate-ensure --check

[ps1noob]

OK, so here is the output from the "./configure..." command:

After that, i processed with the "make..." command and also with the "gdb...." command, which ended like this:

I guess that this is not the expected output?

[martinpitt]

@ps1noob : Nice, you got quite far! In the gdb prompt, type run. That should crash and get you back to the prompt. Then type bt full and copy&paste the output here.

BTW, it would be much easier if you could copy that as text instead of as a screenshot.

Thanks!

[ps1noob]

I did see the crash and here is the (not as a screenshot) output:

`admin@raspberrypi:~/cockpit-310.1 $ gdb --args ./cockpit-certificate-ensure --check GNU gdb (Raspbian 13.1-3) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./cockpit-certificate-ensure... (gdb) run Starting program: /home/admin/cockpit-310.1/cockpit-certificate-ensure --check [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1". Unable to find any certificate file Would create a self-signed certificate [Inferior 1 (process 738) exited with code 01] (gdb) bt full No stack. (gdb)`

[martinpitt]

Hmm, it actually didn't crash here. But just to confirm, if you run /usr/lib/cockpit/cockpit-certificate-ensure --check, it still crashes? @jack60612 ran it under gdb above, but that was someone else. Can you please check if it crashes under gdb:

gdb --args /usr/lib/cockpit/cockpit-certificate-ensure --check`

If that crashes, that's a fun bug then -- the technical :wink: term is Heisenbug. The recipe above rebuilt the source with debugging enabled and compiler optimizations disabled, which means that it probably only happens without them. In that case it makes sense to repeat this exercise without configuring with --enable-debug, but let's do that step by step after confirmation (Debian builds with more compiler options).

[ps1noob]

Nice to learn something new, thanks! 😊

Well, i think i did everything as needed: `admin@raspberrypi:~ $ /usr/lib/cockpit/cockpit-certificate-ensure --check Segmentation fault admin@raspberrypi:~ $ gdb --args /usr/lib/cockpit/cockpit-certificate-ensure --check GNU gdb (Raspbian 13.1-3) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/lib/cockpit/cockpit-certificate-ensure... (No debugging symbols found in /usr/lib/cockpit/cockpit-certificate-ensure) (gdb) run Starting program: /usr/lib/cockpit/cockpit-certificate-ensure --check [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault. 0x00450722 in ?? () (gdb) bt full

0 0x00450722 in ?? ()

No symbol table info available.

1 0x0040156e in ?? ()

No symbol table info available. Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)`

[martinpitt]

OK, thanks for confirming! Now we need to become a bit creative. Let's first try to build without debug mode:

make distclean
./configure --prefix=/usr --with-cockpit-user=cockpit-ws --with-cockpit-ws-instance-user=cockpit-wsinstance --sysconfdir=/etc --localstatedir=/var --enable-strict
make -j$(nproc)
gdb --args ./cockpit-certificate-ensure --check

and in gdb, again, run and if it crashes, bt full. If it doesn't crash, then we need to tweak the compiler flags even more. D'oh..

[martinpitt]

if that also doesn't crash, then please try something else: rebuild the Debian package locally, and install it, and see if it still crashes -- it may not be the compiler flags, but the mere act of rebuilding against the current version of GnuTLS:

make distclean
dpkg-buildpackage -us -uc -b
sudo dpkg -i ../cockpit-ws*.deb

Then again, check if it crashes:

/usr/lib/cockpit/cockpit-certificate-ensure --check

based on the outcome I'll then think about a next step.

If the previous step (rebuilding without debug) does crash, then the above isn't necessary.

[ps1noob]

OK, thanks for confirming! Now we need to become a bit creative. Let's first try to build without debug mode:

make distclean
./configure --prefix=/usr --with-cockpit-user=cockpit-ws --with-cockpit-ws-instance-user=cockpit-wsinstance --sysconfdir=/etc --localstatedir=/var --enable-strict
make -j$(nproc)
gdb --args ./cockpit-certificate-ensure --check

and in gdb, again, run and if it crashes, bt full. If it doesn't crash, then we need to tweak the compiler flags even more. D'oh..

So, this one looks like this:

admin@raspberrypi:~/cockpit-310.1 $ gdb --args ./cockpit-certificate-ensure --check
GNU gdb (Raspbian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./cockpit-certificate-ensure...
(gdb) run
Starting program: /home/admin/cockpit-310.1/cockpit-certificate-ensure --check
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Unable to find any certificate file
Would create a self-signed certificate
[Inferior 1 (process 3272) exited with code 01]
(gdb) bt full
No stack.
(gdb)

So, as you told me before, that shouldn't be a crash.

if that also doesn't crash, then please try something else: rebuild the Debian package locally, and install it, and see if it still crashes -- it may not be the compiler flags, but the mere act of rebuilding against the current version of GnuTLS:

make distclean
dpkg-buildpackage -us -uc -b
sudo dpkg -i ../cockpit-ws*.deb

Then again, check if it crashes:

/usr/lib/cockpit/cockpit-certificate-ensure --check

based on the outcome I'll then think about a next step.

If the previous step (rebuilding without debug) does crash, then the above isn't necessary.

I can't even get to this point...

.....................
make  install-exec-hook
make[5]: Entering directory '/home/admin/cockpit-310.1'
chown -f root:cockpit-wsinstance /home/admin/cockpit-310.1/debian/tmp/usr/lib/cockpit/cockpit-session || true
test "cockpit-ws" != "root" && chmod -f 4750 /home/admin/cockpit-310.1/debian/tmp/usr/lib/cockpit/cockpit-session || true
mkdir -p /home/admin/cockpit-310.1/debian/tmp/etc/motd.d
ln -sTfr /home/admin/cockpit-310.1/debian/tmp/run/cockpit/motd /home/admin/cockpit-310.1/debian/tmp/etc/motd.d/cockpit
mkdir -p /home/admin/cockpit-310.1/debian/tmp/etc/issue.d
ln -sTfr /home/admin/cockpit-310.1/debian/tmp/run/cockpit/motd /home/admin/cockpit-310.1/debian/tmp/etc/issue.d/cockpit.issue
mkdir -p /home/admin/cockpit-310.1/debian/tmp/etc/cockpit/ws-certs.d /home/admin/cockpit-310.1/debian/tmp/etc/cockpit/machines.d
chmod 755 /home/admin/cockpit-310.1/debian/tmp/etc/cockpit/ws-certs.d /home/admin/cockpit-310.1/debian/tmp/etc/cockpit/machines.d
make[5]: Leaving directory '/home/admin/cockpit-310.1'
 /usr/bin/mkdir -p '/home/admin/cockpit-310.1/debian/tmp/usr/lib/cockpit'
  /usr/bin/install -c cockpit-ws '/home/admin/cockpit-310.1/debian/tmp/usr/lib/cockpit'
mkdir -p /home/admin/cockpit-310.1/debian/tmp/usr/share/doc/cockpit
/usr/bin/install -c -m 644 doc/guide/html/* /home/admin/cockpit-310.1/debian/tmp/usr/share/doc/cockpit
cd ./dist; find */* -type f -exec install -D -m 644 '{}' '/home/admin/cockpit-310.1/debian/tmp/usr/share/cockpit/{}' \;
python3 -m pip install --no-index --force-reinstall --root='/home/admin/cockpit-310.1/debian/tmp/' --prefix='/usr' \
        "$(python3 '.'/src/build_backend.py --wheel '.' tmp/wheel)"
Processing ./tmp/wheel/cockpit-310.1-py3-none-any.whl
Installing collected packages: cockpit
  Attempting uninstall: cockpit
    Found existing installation: cockpit 311
    Uninstalling cockpit-311:
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/bin/cockpit-bridge'
Consider using the `--user` option or check the permissions.

make[4]: *** [Makefile:9025: install-python] Error 1
make[4]: Leaving directory '/home/admin/cockpit-310.1'
make[3]: *** [Makefile:8168: install-am] Error 2
make[3]: Leaving directory '/home/admin/cockpit-310.1'
make[2]: *** [Makefile:8161: install] Error 2
make[2]: Leaving directory '/home/admin/cockpit-310.1'
dh_auto_install: error: make -j1 install DESTDIR=/home/admin/cockpit-310.1/debian/tmp AM_UPDATE_INFO_DIR=no returned exit code 2
make[1]: *** [debian/rules:29: override_dh_auto_install] Error 2
make[1]: Leaving directory '/home/admin/cockpit-310.1'
make: *** [debian/rules:17: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
admin@raspberrypi:~/cockpit-310.1 $ sudo dpkg -i ../cockpit-ws*.deb
dpkg: error: cannot access archive '../cockpit-ws*.deb': No such file or directory

[martinpitt]

Hmm, apparently there was an unclean build tree. Can you retry that again, i.e. delete the build tree and rebuild the Debian source package?

cd ..
rm -rf cockpit-*
dpkg-source -x cockpit*.dsc
cd cockpit-*
dpkg-buildpackage -us -uc -b
sudo dpkg -i ../cockpit-ws*.deb

[ps1noob]

Sure thing! Unfortunately, it looks like this (end of the dpkg-buildpackage process and the following processes):

................................
Installing collected packages: cockpit
  Attempting uninstall: cockpit
    Found existing installation: cockpit 311
    Uninstalling cockpit-311:
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/bin/cockpit-bridge'
Consider using the `--user` option or check the permissions.

make[4]: *** [Makefile:9025: install-python] Error 1
make[4]: Leaving directory '/home/admin/cockpit-310.1'
make[3]: *** [Makefile:8168: install-am] Error 2
make[3]: Leaving directory '/home/admin/cockpit-310.1'
make[2]: *** [Makefile:8161: install] Error 2
make[2]: Leaving directory '/home/admin/cockpit-310.1'
dh_auto_install: error: make -j1 install DESTDIR=/home/admin/cockpit-310.1/debian/tmp AM_UPDATE_INFO_DIR=no returned exit code 2
make[1]: *** [debian/rules:29: override_dh_auto_install] Error 2
make[1]: Leaving directory '/home/admin/cockpit-310.1'
make: *** [debian/rules:17: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
admin@raspberrypi:~/cockpit-310.1 $ sudo dpkg -i ../cockpit-ws*.deb
dpkg: error: cannot access archive '../cockpit-ws*.deb': No such file or directory
admin@raspberrypi:~/cockpit-310.1 $ ls -la ./
total 19600
drwxr-xr-x 19 admin admin    4096 Feb 26 19:59 .
drwx------  3 admin admin    4096 Feb 26 19:32 ..
-rw-r--r--  1 admin admin   69344 Feb 26 19:33 aclocal.m4
-rw-r--r--  1 admin admin     217 Feb  2 15:16 AUTHORS
drwxr-xr-x  2 admin admin    4096 Feb 26 19:34 autom4te.cache
-rwxr-xr-x  1 admin admin    9200 Feb  2 15:16 build.js
-rwxr-xr-x  1 admin admin   65508 Feb 26 19:44 cockpit-certificate-ensure
-rwxr-xr-x  1 admin admin  690320 Feb 26 19:43 cockpit-pcp
-rwxr-xr-x  1 admin admin  155956 Feb 26 19:43 cockpit-session
-rwxr-xr-x  1 admin admin  384824 Feb 26 19:43 cockpit-ssh
-rwxr-xr-x  1 admin admin  157364 Feb 26 19:44 cockpit-tls
-rwxr-xr-x  1 admin admin 1002784 Feb 26 19:41 cockpit-ws
-rwxr-xr-x  1 admin admin   30588 Feb 26 19:44 cockpit-wsinstance-factory
-rw-r--r--  1 admin admin    5836 Feb 26 19:35 config.h
-rw-r--r--  1 admin admin    5394 Feb 26 19:34 config.h.in
-rw-r--r--  1 admin admin    5394 Feb  2 15:16 config.h.in~
-rw-r--r--  1 admin admin   38525 Feb 26 19:35 config.log
-rwxr-xr-x  1 admin admin   38884 Feb 26 19:35 config.status
-rwxr-xr-x  1 admin admin  271901 Feb 26 19:33 configure
-rwxr-xr-x  1 admin admin  271901 Feb  2 15:16 configure~
.................................

[martinpitt]

@ps1noob Ah meh, apparently you found an issue with pip -- it's apparently not possible to build the cockpit source package if the cockpit-bridge package is already installed (we never notice because both Debian and our CI build the debs in an isolated environment). We'll have to find some time to investigate a fix / magic option to pip. In the meantime you could work around this by temporarily doing

sudo dpkg -P --force-depends cockpit-bridge

then building the package as above, and installing the bridge again at the end:

sudo dpkg -i .../cockpit-bridge*.deb ./cockpit-ws*.deb

[martinpitt]

Reproduced the build failure in a debian testing container with cockpit-bridge installed. In upstream git checkout:

./autogen.sh --prefix=/usr --with-cockpit-user=cockpit-ws --with-cockpit-ws-instance-user=cockpit-wsinstance --sysconfdir=/etc --localstatedir=/var --enable-strict --enable-debug
make -j$(nproc)
make install DESTDIR=/tmp/i

Fails with

Processing ./tmp/wheel/cockpit-0-py3-none-any.whl
Installing collected packages: cockpit
  Attempting uninstall: cockpit
    Found existing installation: cockpit 311
ERROR: Cannot uninstall cockpit 311, RECORD file not found. Hint: The package was installed by debian.

After that, it can be reproduced more directly with

python3 -m pip install --no-index --force-reinstall --root=/tmp/i/ --prefix=/usr "$(python3 ./src/build_backend.py --wheel . tmp/wheel)"

This is very clearly a pip bug -- with --root it should entirely ignore anything outside of /tmp/i. See https://github.com/pypa/pip/issues/3063 . Fortunately there is a simple workaround, add --ignore-installed. I sent PR #20103 to do that.

In the meantime, instead of the sudo dpkg -P --force-depends from above (which is a rather dramatic and a bit dangerous hammer), instead do this, inside of the cockpit-* build tree that previously failed:

sed -i 's/--force-reinstall/& --ignore-installed/' src/Makefile.am
dpkg-buildpackage -us -uc -b -j4 -nc

This should continue the previously aborted build, and succeed. After that, please continue with

sudo dpkg -i ../cockpit-ws*.deb
gdb --args /usr/lib/cockpit/cockpit-certificate-ensure --check

If that works, then a mere package rebuild helps, and that's some weird toolchain/gnutls ABI break/etc. issue, and I'm neither qualified enough nor do I have the time to track that down. But at least you have a working installation, and the next Debian update should fix it (in a few days, when 312 gets out).

If it still crashes, can you please confirm that calling ./cockpit-certificate-ensure --check (i.e. out of the build tree) still succeeds?

[ps1noob]

Well.... that's some very good experience with someone developing software, thank you! Not my usual business, neither my usual experience with developers, but here we are 😎

Here is the output of the commands from your last post, they seem fine to me.

..............................
dpkg-deb: building package 'cockpit' in '../cockpit_310.1-1~bpo12+1_all.deb'.
dpkg-deb: building package 'cockpit-networkmanager' in '../cockpit-networkmanager_310.1-1~bpo12+1_all.deb'.
dpkg-deb: building package 'cockpit-storaged' in '../cockpit-storaged_310.1-1~bpo12+1_all.deb'.
dpkg-deb: building package 'cockpit-ws' in '../cockpit-ws_310.1-1~bpo12+1_armhf.deb'.
dpkg-deb: building package 'cockpit-bridge' in '../cockpit-bridge_310.1-1~bpo12+1_armhf.deb'.
dpkg-deb: building package 'cockpit-pcp' in '../cockpit-pcp_310.1-1~bpo12+1_armhf.deb'.
dpkg-deb: building package 'cockpit-system' in '../cockpit-system_310.1-1~bpo12+1_all.deb'.
dpkg-deb: building package 'cockpit-bridge-dbgsym' in '../cockpit-bridge-dbgsym_310.1-1~bpo12+1_armhf.deb'.
dpkg-deb: building package 'cockpit-pcp-dbgsym' in '../cockpit-pcp-dbgsym_310.1-1~bpo12+1_armhf.deb'.
dpkg-deb: building package 'cockpit-ws-dbgsym' in '../cockpit-ws-dbgsym_310.1-1~bpo12+1_armhf.deb'.
dpkg-deb: building package 'cockpit-doc' in '../cockpit-doc_310.1-1~bpo12+1_all.deb'.
dpkg-deb: building package 'cockpit-packagekit' in '../cockpit-packagekit_310.1-1~bpo12+1_all.deb'.
dpkg-deb: building package 'cockpit-sosreport' in '../cockpit-sosreport_310.1-1~bpo12+1_all.deb'.
dpkg-deb: building package 'cockpit-tests' in '../cockpit-tests_310.1-1~bpo12+1_armhf.deb'.
dpkg-deb: building package 'cockpit-tests-dbgsym' in '../cockpit-tests-dbgsym_310.1-1~bpo12+1_armhf.deb'.
 dpkg-genbuildinfo --build=binary -O../cockpit_310.1-1~bpo12+1_armhf.buildinfo
 dpkg-genchanges --build=binary -O../cockpit_310.1-1~bpo12+1_armhf.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-source: info: using options from cockpit-310.1/debian/source/options: --extend-diff-ignore=src/ssh/mock_rsa_key.pub$
dpkg-buildpackage: info: binary-only upload (no source included)
admin@raspberrypi:~/cockpit-310.1 $ sudo dpkg -i ../cockpit-ws*.deb
dpkg: warning: downgrading cockpit-ws from 311-1~bpo12+1 to 310.1-1~bpo12+1
(Reading database ... 98349 files and directories currently installed.)
Preparing to unpack .../cockpit-ws_310.1-1~bpo12+1_armhf.deb ...
Unpacking cockpit-ws (310.1-1~bpo12+1) over (311-1~bpo12+1) ...
Selecting previously unselected package cockpit-ws-dbgsym.
Preparing to unpack .../cockpit-ws-dbgsym_310.1-1~bpo12+1_armhf.deb ...
Unpacking cockpit-ws-dbgsym (310.1-1~bpo12+1) ...
Setting up cockpit-ws (310.1-1~bpo12+1) ...
Adjusting /usr/lib/cockpit/cockpit-session permissions...
Setting up cockpit-ws-dbgsym (310.1-1~bpo12+1) ...
Processing triggers for man-db (2.11.2-2) ...
admin@raspberrypi:~/cockpit-310.1 $ gdb --args /usr/lib/cockpit/cockpit-certificate-ensure --check
GNU gdb (Raspbian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/cockpit/cockpit-certificate-ensure...
Reading symbols from /usr/lib/debug/.build-id/99/b394f0c4a2bfedc43bcbcc7f3ad0477b95e3bb.debug...
(gdb) run
Starting program: /usr/lib/cockpit/cockpit-certificate-ensure --check
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Unable to find any certificate file
Would create a self-signed certificate
[Inferior 1 (process 5505) exited with code 01]
(gdb) bt full
No stack.
(gdb)

I can also confirm that ./cockpit-certificate-ensure --check runs as it should (i think? 😅):

admin@raspberrypi:~/cockpit-310.1 $ ./cockpit-certificate-ensure --check
Unable to find any certificate file
Would create a self-signed certificate

As i am willing to wait for cockpit version 312 (in hope of getting the fix in the official build), i would provision my sd card with a fresh install of Raspberry Pi OS and check if cockpit 312 will be the version that i was looking for 😎

[martinpitt]

OK, thanks @ps1noob for bearing with me, that was a bit of an odyssey :sweat_smile:

So you have that locally built cockpit-ws*.deb now which you can copy around, and in a few days 312 should be available. Please let us know if it happens again with that version.

I think there's nothing actionable left here.