Closed castilma closed 6 months ago
The message ERROR Could not init PAM account management for user 'root': Module is unknown
should be more verbose, i.e. state which module could not be loaded.
2022-02-15 23:12:17 ERROR Could not read job pid because of closed pipe: setting it to -1
2022-02-15 23:42:12 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 1): Success
To debug this, it would be a start to find out, how the pipe could have been closed. And, why read_write_pipe seemingly tries to read from a closed pipe/invalid file descriptor(?) returning 0 bytes instead of an error. If it reads from somewhere else, it would be good to know why that failed.
Today I noticed the error again, this time on my laptop, but with the same os.
2022-08-03 00:01:00 INFO Job '/usr/bin/run-parts /etc/cron.hourly' started for user systab (pid 7316)
2022-08-03 00:01:00 INFO Job '/usr/bin/run-parts /etc/cron.hourly' completed
2022-08-03 00:02:00 INFO Job '/usr/bin/run-parts /etc/cron.daily' started for user systab (pid 7417)
2022-08-03 00:03:11 INFO Job '/usr/bin/run-parts /etc/cron.daily' completed
2022-08-03 01:01:00 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 1): Success
2022-08-03 01:01:01 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 2): Success
2022-08-03 01:01:02 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 3): Success
2022-08-03 01:01:03 ERROR Could not read job pid because of closed pipe: setting it to -1
2022-08-03 11:37:09 INFO suspend/hibernate detected: we woke up after 37999s instead of 1800s. The system was suspended for 36927s.
2022-08-03 11:37:09 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 1): Success
2022-08-03 11:37:10 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 2): Success
2022-08-03 11:37:11 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 3): Success
2022-08-03 11:37:12 ERROR Could not read job pid because of closed pipe: setting it to -1
2022-08-03 12:01:00 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 1): Success
2022-08-03 12:01:01 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 2): Success
2022-08-03 12:01:02 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 3): Success
2022-08-03 12:01:03 ERROR Could not read job pid because of closed pipe: setting it to -1
2022-08-03 13:01:00 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 1): Success
2022-08-03 13:01:01 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 2): Success
2022-08-03 13:01:02 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 3): Success
2022-08-03 13:01:03 ERROR Could not read job pid because of closed pipe: setting it to -1
2022-08-03 14:01:00 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 1): Success
2022-08-03 14:01:01 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 2): Success
2022-08-03 14:01:02 ERROR read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 3): Success
2022-08-03 14:01:03 ERROR Could not read job pid because of closed pipe: setting it to -1
2022-08-03 14:13:41 INFO suspend/hibernate detected: we woke up after 392s instead of 1800s. The system was suspended for 168s.
and theres been a system upgrade prior to the error:
[2022-08-03T00:40:35+0200] [ALPM] running '60-mkinitcpio-remove.hook'...
[2022-08-03T00:40:35+0200] [ALPM] transaction started
[2022-08-03T00:40:36+0200] [ALPM] upgraded linux-api-headers (5.17.5-2 -> 5.18.15-1)
[2022-08-03T00:40:38+0200] [ALPM] warning: /etc/locale.gen installed as /etc/locale.gen.pacnew
[2022-08-03T00:40:38+0200] [ALPM] upgraded glibc (2.35-6 -> 2.36-1)
[2022-08-03T00:40:38+0200] [ALPM-SCRIPTLET] Generating locales...
[2022-08-03T00:40:41+0200] [ALPM-SCRIPTLET] de_DE.UTF-8... done
[2022-08-03T00:40:41+0200] [ALPM-SCRIPTLET] Generation complete.
[2022-08-03T00:40:43+0200] [ALPM] upgraded gcc-libs (12.1.0-3 -> 12.1.1-4)
[2022-08-03T00:40:43+0200] [ALPM] upgraded lib32-glibc (2.35-6 -> 2.36-1)
[2022-08-03T00:40:43+0200] [ALPM] upgraded libelf (0.187-1 -> 0.187-2)
[2022-08-03T00:40:44+0200] [ALPM] upgraded binutils (2.38-6 -> 2.38-7)
[2022-08-03T00:40:44+0200] [ALPM] upgraded mpfr (4.1.0.p13-2 -> 4.1.0.p13-3)
[2022-08-03T00:41:20+0200] [ALPM] upgraded linux (5.18.14.artix1-1 -> 5.18.15.artix1-2)
[2022-08-03T00:41:20+0200] [ALPM] upgraded acpi_call (1.2.2-57 -> 1.2.2-60)
[2022-08-03T00:41:21+0200] [ALPM] upgraded elfutils (0.187-1 -> 0.187-2)
[2022-08-03T00:41:21+0200] [ALPM] upgraded libsysprof-capture (3.44.0-1 -> 3.44.0-2)
[2022-08-03T00:41:21+0200] [ALPM] upgraded gsm (1.0.20-1 -> 1.0.21-1)
[2022-08-03T00:41:21+0200] [ALPM] upgraded opencore-amr (0.1.5-5 -> 0.1.6-1)
[2022-08-03T00:41:23+0200] [ALPM] upgraded libtiff (4.4.0-1 -> 4.4.0-3)
[2022-08-03T00:41:32+0200] [ALPM] upgraded firefox (103.0-1 -> 103.0.1-1)
[2022-08-03T00:41:32+0200] [ALPM] upgraded firefox-i18n-de (103.0-1 -> 103.0.1-1)
[2022-08-03T00:41:36+0200] [ALPM] upgraded gcc (12.1.0-3 -> 12.1.1-4)
[2022-08-03T00:41:52+0200] [ALPM] upgraded gcc-fortran (12.1.0-3 -> 12.1.1-4)
[2022-08-03T00:42:23+0200] [ALPM] upgraded libtool (2.4.7-2 -> 2.4.7-3)
[2022-08-03T00:42:26+0200] [ALPM] upgraded imagemagick (7.1.0.44-1 -> 7.1.0.45-1)
[2022-08-03T00:42:27+0200] [ALPM] upgraded iptables (1:1.8.8-1 -> 1:1.8.8-2)
[2022-08-03T00:42:30+0200] [ALPM] upgraded jre8-openjdk-headless (8.342.u07-1 -> 8.345.u01-1)
[2022-08-03T00:42:31+0200] [ALPM] upgraded lib32-gcc-libs (12.1.0-3 -> 12.1.1-4)
[2022-08-03T00:42:31+0200] [ALPM] upgraded lib32-libxcrypt (4.4.28-1 -> 4.4.28-2)
[2022-08-03T00:42:42+0200] [ALPM] upgraded libreoffice-fresh (7.3.4-2 -> 7.3.5-1)
[2022-08-03T00:42:52+0200] [ALPM] upgraded libreoffice-fresh-de (7.3.4-1 -> 7.3.5-1)
[2022-08-03T00:42:52+0200] [ALPM] upgraded libwpe (1.12.0-2 -> 1.12.2-1)
[2022-08-03T00:43:05+0200] [ALPM] upgraded sddm (0.19.0-8 -> 0.19.0-8.1)
[2022-08-03T00:43:05+0200] [ALPM] upgraded tp_smapi (0.43-410 -> 0.43-413)
[2022-08-03T00:43:05+0200] [ALPM] upgraded wavpack (5.4.0-1 -> 5.5.0-1)
[2022-08-03T00:43:05+0200] [ALPM] upgraded wayland-protocols (1.25-1 -> 1.26-1)
[2022-08-03T00:43:06+0200] [ALPM] upgraded webkit2gtk (2.36.5-1 -> 2.36.5-2)
[2022-08-03T00:43:07+0200] [ALPM] upgraded webkit2gtk-4.1 (2.36.5-1 -> 2.36.5-2)
[2022-08-03T00:43:08+0200] [ALPM] transaction completed
[2022-08-03T00:43:09+0200] [ALPM] running '20-sysusers.hook'...
[2022-08-03T00:43:10+0200] [ALPM] running '30-tmpfiles.hook'...
[2022-08-03T00:43:10+0200] [ALPM] running '30-update-mime-database.hook'...
[2022-08-03T00:43:11+0200] [ALPM] running '60-depmod.hook'...
[2022-08-03T00:45:48+0200] [ALPM] running '90-mkinitcpio-install.hook'...
[2022-08-03T00:45:49+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2022-08-03T00:45:49+0200] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2022-08-03T00:45:49+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.18.15-artix1-2
[2022-08-03T00:45:49+0200] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2022-08-03T00:45:50+0200] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2022-08-03T00:45:51+0200] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2022-08-03T00:45:51+0200] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2022-08-03T00:45:52+0200] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2022-08-03T00:45:52+0200] [ALPM-SCRIPTLET] -> Running build hook: [consolefont]
[2022-08-03T00:45:52+0200] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2022-08-03T00:45:52+0200] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2022-08-03T00:45:52+0200] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2022-08-03T00:45:53+0200] [ALPM-SCRIPTLET] -> Running build hook: [encrypt]
[2022-08-03T00:45:54+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qat_4xxx
[2022-08-03T00:45:58+0200] [ALPM-SCRIPTLET] -> Running build hook: [lvm2]
[2022-08-03T00:46:01+0200] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2022-08-03T00:46:01+0200] [ALPM-SCRIPTLET] -> Running build hook: [resume]
[2022-08-03T00:46:01+0200] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2022-08-03T00:46:01+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-08-03T00:46:01+0200] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
[2022-08-03T00:46:02+0200] [ALPM-SCRIPTLET] ==> Image generation successful
[2022-08-03T00:46:02+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2022-08-03T00:46:02+0200] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2022-08-03T00:46:02+0200] [ALPM-SCRIPTLET] ==> Starting build: 5.18.15-artix1-2
[2022-08-03T00:46:02+0200] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2022-08-03T00:46:02+0200] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2022-08-03T00:46:03+0200] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2022-08-03T00:46:03+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2022-08-03T00:46:04+0200] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2022-08-03T00:46:04+0200] [ALPM-SCRIPTLET] -> Running build hook: [consolefont]
[2022-08-03T00:46:04+0200] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2022-08-03T00:46:04+0200] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2022-08-03T00:46:04+0200] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2022-08-03T00:46:07+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla2xxx
[2022-08-03T00:46:07+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2022-08-03T00:46:08+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qed
[2022-08-03T00:46:09+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2022-08-03T00:46:12+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2022-08-03T00:46:13+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla1280
[2022-08-03T00:46:17+0200] [ALPM-SCRIPTLET] -> Running build hook: [encrypt]
[2022-08-03T00:46:17+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qat_4xxx
[2022-08-03T00:46:18+0200] [ALPM-SCRIPTLET] -> Running build hook: [lvm2]
[2022-08-03T00:46:18+0200] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2022-08-03T00:46:20+0200] [ALPM-SCRIPTLET] -> Running build hook: [resume]
[2022-08-03T00:46:20+0200] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2022-08-03T00:46:21+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2022-08-03T00:46:22+0200] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fallback.img
[2022-08-03T00:46:24+0200] [ALPM-SCRIPTLET] ==> Image generation successful
[2022-08-03T00:46:24+0200] [ALPM] running 'detect-old-perl-modules.hook'...
[2022-08-03T00:46:24+0200] [ALPM] running 'gtk-update-icon-cache.hook'...
[2022-08-03T00:46:24+0200] [ALPM] running 'texinfo-install.hook'...
[2022-08-03T00:46:28+0200] [ALPM] running 'update-desktop-database.hook'...
The following packages were update/installed both times:
binutils
elfutils
firefox
firefox-i18n-de
gcc
gcc-fortran
gcc-libs
glibc
imagemagick
lib32-gcc-libs
lib32-glibc
libelf
libsysprof-capture
libtool
linux
linux-api-headers
mpfr
webkit2gtk
webkit2gtk-4.1
Before the error printing code is a comment, asking, whether writing to a closed pipe is really an error, and that it should never happen. (job.c). Writing to a closed pipe is indicative of an error, and it can happen. The 'real' errror message is available in the mail or syslog:
cron.notice: Aug 3 00:03:11 fcron[7416]: Job '/usr/bin/run-parts /etc/cron.daily' completed
cron.err: Aug 3 01:01:00 fcron[18828]: Could not init PAM account management for user 'root': Module is unknown
cron.err: Aug 3 01:01:00 fcron[18828]: Job '/usr/bin/run-parts /etc/cron.hourly' has *not* run.
cron.err: Aug 3 01:01:00 fcron[18828]: Couldn't exec '/usr/bin/sendmail': No such file or directory
cron.err: Aug 3 01:01:00 fcron[18828]: fcron child aborted: this does not affect the main fcron daemon, but this may prevent a job from being run or an email from being sent.
cron.err: Aug 3 01:01:00 fcron[1015]: read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 1): Success
cron.err: Aug 3 01:01:01 fcron[1015]: read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 2): Success
cron.err: Aug 3 01:01:02 fcron[1015]: read_write_pipe(): read/write returned 0: retrying... (size: 4, size_processed: 0, num_retry: 3): Success
cron.err: Aug 3 01:01:03 fcron[1015]: Could not read job pid because of closed pipe: setting it to -1
Here are the open files of fcron
lr-x------ 1 root root 64 8. Aug 14:31 0 -> /dev/null
l-wx------ 1 root root 64 8. Aug 14:31 1 -> 'pipe:[17223]'
l-wx------ 1 root root 64 8. Aug 14:31 2 -> 'pipe:[17223]'
lrwx------ 1 root root 64 8. Aug 14:31 3 -> /run/fcron.pid
lrwx------ 1 root root 64 8. Aug 14:31 4 -> 'socket:[17652]'
lrwx------ 1 root root 64 8. Aug 14:31 5 -> 'socket:[17653]'
lrwx------ 1 root root 64 8. Aug 14:31 6 -> 'anon_inode:[timerfd]'
and an strace:
pselect6(7, [5 6], NULL, NULL, {tv_sec=49, tv_nsec=223536660}, NULL) = 0 (Timeout)
pipe2([7, 8], 0) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb77d372e90) = 14791
close(8) = 0
read(7, "", 4) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=14791, si_uid=1000, si_status=2, si_utime=0, si_stime=0} ---
rt_sigreturn({mask=[]}) = 0
getpid() = 1015
sendto(4, "<75>Aug 8 15:07:00 fcron[1015]:"..., 138, MSG_NOSIGNAL, NULL, 0) = 138
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=2298, ...}, 0) = 0
write(2, "2022-08-08 15:07:00 ERROR ", 26) = 26
write(2, "read_write_pipe(): read/write re"..., 106) = 106
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda95bf740) = 0
read(7, "", 4) = 0
getpid() = 1015
sendto(4, "<75>Aug 8 15:07:01 fcron[1015]:"..., 138, MSG_NOSIGNAL, NULL, 0) = 138
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=2298, ...}, 0) = 0
write(2, "2022-08-08 15:07:01 ERROR ", 26) = 26
write(2, "read_write_pipe(): read/write re"..., 106) = 106
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda95bf740) = 0
read(7, "", 4) = 0
getpid() = 1015
sendto(4, "<75>Aug 8 15:07:02 fcron[1015]:"..., 138, MSG_NOSIGNAL, NULL, 0) = 138
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=2298, ...}, 0) = 0
write(2, "2022-08-08 15:07:02 ERROR ", 26) = 26
write(2, "read_write_pipe(): read/write re"..., 106) = 106
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffda95bf740) = 0
read(7, "", 4) = 0
getpid() = 1015
sendto(4, "<75>Aug 8 15:07:03 fcron[1015]:"..., 96, MSG_NOSIGNAL, NULL, 0) = 96
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=2298, ...}, 0) = 0
write(2, "2022-08-08 15:07:03 ERROR ", 26) = 26
write(2, "Could not read job pid because o"..., 64) = 64
close(7) = 0
now I looked at syslog again:
authpriv.err: Aug 3 01:01:00 fcron[18828]: PAM unable to dlopen(/usr/lib/security/pam_unix.so): /usr/lib/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by /usr/lib/libpthread.so.0)
authpriv.err: Aug 3 01:01:00 fcron[18828]: PAM adding faulty module: /usr/lib/security/pam_unix.so
I guess it's the glibc update which brakes the ABI of something?
So, is this an fcron bug? I never experienced any daemon problems with PAM after an update. So maybe fcron is doing something more vulnerable. But I'm not sure how many daemons I'm using, that use PAM...
I wonder who's at fault: An update causes already running processes to continue running with an outdated libc, which breaks dlopen with newer libraries (I think that's the problem).
There is an archlinux forum post with a similar error message. Also, restarting fcron fixes it. So, even though the problem doesn't seem to be a fcron bug, maybe fcron should restart itself to fix this.
@yo8192 Could you please mention the commit, that fixed this?
Two weeks ago I only find this, which seems to fix #23, but not this issue.
@castilma I did a clean-up of issues then and I assumed this one was old and probably no longer relevant. I should have put a comment to say so, apologies for the confusion.
What you described here looks more like a distribution issue than a fcron issue. Also no one else has reported this on other distributions.
I see fcron is packaged under Artix Linux: https://packages.artixlinux.org/details/fcron. Did you use the distribution package to install fcron? If so I think it would be best to contact the Artix package maintainer to investigate and possibly to add some auto-restart in that scenario if that seems to help.
Also your original message mentions fcron 3.2.1... have you seen this problem with the latest fcron version 3.3.1?
I installed the one distributed by artix. No, I haven't seen it since.
Recently I noticed fcron to be unable to run jobs again, and I had to restart it.
I get error mails (without subject) about the failed jobs. Here are 2 examples:
The only thing that might correlate is a system update I ran before:
I'm on artixlinux with runit. I should have older logs from another time this happened. Tell me, if you need them.