Can't launch glusterfs after upgrading glusterfs from 1:9.3-2 to 1:10.1-1

[PisoMojado]

Description of problem: GlusterFS will not launch on one of my peers after system upgrade.

The exact command to reproduce the issue: sudo systemctl start glusterd

The full output of the command that failed:

× glusterd.service - GlusterFS, a clustered file-system server
     Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Mon 2022-06-20 14:08:01 CDT; 10min ago
       Docs: man:glusterd(8)
    Process: 1501 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=1/FAILURE)
        CPU: 131ms

Jun 20 14:08:00 icarus glusterd[1502]: llistxattr 1
Jun 20 14:08:00 icarus glusterd[1502]: setfsid 1
Jun 20 14:08:00 icarus glusterd[1502]: epoll.h 1
Jun 20 14:08:00 icarus glusterd[1502]: xattr.h 1
Jun 20 14:08:00 icarus glusterd[1502]: st_atim.tv_nsec 1
Jun 20 14:08:00 icarus glusterd[1502]: package-string: glusterfs 10.1
Jun 20 14:08:00 icarus glusterd[1502]: ---------
Jun 20 14:08:01 icarus systemd[1]: glusterd.service: Control process exited, code=exited, status=1/FAILURE
Jun 20 14:08:01 icarus systemd[1]: glusterd.service: Failed with result 'exit-code'.
Jun 20 14:08:01 icarus systemd[1]: Failed to start GlusterFS, a clustered file-system server.

Expected results: glusterfs launches

Mandatory info: - The output of the gluster volume info command: Connection failed. Please check if gluster daemon is operational. - The output of the gluster volume status command: Connection failed. Please check if gluster daemon is operational. - The output of the gluster volume heal command: Connection failed. Please check if gluster daemon is operational. - Provide logs present on following locations of client and server nodes: /var/log/glusterfs/glusterd.log --------- [2022-06-20 19:08:00.613860 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterd: Started running version [{arg=/usr/sbin/glusterd}, {version=10.1}, {cmdlinestr=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO}] [2022-06-20 19:08:00.615423 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 1502 [2022-06-20 19:08:00.618391 +0000] W [MSGID: 101249] [gf-io-uring.c:408:gf_io_uring_setup] 0-io: Kernel's I/O URing implementation doesn't support given data. [Invalid argument] pending frames: patchset: git://git.gluster.org/glusterfs.git signal received: 4 time of crash: 2022-06-20 19:08:00 +0000 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 10.1 /usr/lib/libglusterfs.so.0(+0x231bc)[0xb6e731bc] /usr/lib/libglusterfs.so.0(gf_print_trace+0x5d8)[0xb6e7bf6c] /usr/lib/libc.so.6(__default_sa_restorer+0x0)[0xb6bf5d10] /usr/lib/libglusterfs.so.0(gf_io_run+0x73c)[0xb6ed972c] /usr/sbin/glusterd(main+0x814)[0x4c3e74] /usr/lib/libc.so.6(+0x1d9dc)[0xb6bdd9dc] /usr/lib/libc.so.6(__libc_start_main+0xa4)[0xb6bddaf0] ---------

**- Is there any crash ? Provide the backtrace and coredump coredump: `Jun 20 14:08:01 icarus systemd-coredump[1506]: [🡕] Process 1502 (glusterd) of user 0 dumped core.

                                           Module linux-vdso.so.1 with build-id 9d402c7878305288a5b8e36a94e577ebe408c502
                                           Module libresolv.so.2 with build-id 7b3060a70556f8aaab35c2213a5a0c147f647b73
                                           Module libkeyutils.so.1 with build-id 80652a048f08ee7d77823d362ad45290e05282a5
                                           Module libkrb5support.so.0 with build-id 3491ec159e0cccbd43d914449eb4d9802efddd1a
                                           Module libcom_err.so.2 with build-id d3993375f9c120f105de38198c4c494fa5191a53
                                           Module libk5crypto.so.3 with build-id 746de6643b9cb1e32cf93521eff9a05cf1478f58
                                           Module libkrb5.so.3 with build-id 6b0bf73ac2835c76c382e551327be8b6d5f72784
                                           Module libstdc++.so.6 with build-id 175c218d382097ad38dbbaf2ab615dd39f641b1b
                                           Module libpthread.so.0 with build-id 0b0422739722054f65f9f78c4ac441ebc21cd01e
                                           Module libgssapi_krb5.so.2 with build-id 10f3114c7a7946178a56bc45960b24ad8a2171e0
                                           Module libtcmalloc_minimal.so.4 with build-id 6b8e56870859bfb776a6269b62d0f4e0d1879853
                                           Module libcrypto.so.1.1 with build-id 3f4313bab562b8443cdc8b314598a1e510a3b6f6
                                           Module liburcu-cds.so.8 with build-id 19d55ce3878d8db0840a403f753112d041318e75
                                           Module liburcu-common.so.8 with build-id d3975650ef7fcb9736e7df0a4486f65e3a5c4754
                                           Module liburcu-bp.so.8 with build-id 109e5dbb0033d6ba118622fc9f87542558088cba
                                           Module libuuid.so.1 with build-id be65c151afb527ab8d22369276334d6c17c88b53
                                           Module libm.so.6 with build-id 03e814c990762eeb9da12de241a4f42322248e45
                                           Module libz.so.1 with build-id f5e8b23636191e87948dc2c6f3c5fc2f243d9b08
                                           Module ld-linux-armhf.so.3 with build-id 072bb4cd73afd5d62040c7f3f482dbe17719bfea
                                           Module libc.so.6 with build-id ad84e29cae6a8880108cc3a95754d84ca22799e8
                                           Module libgcc_s.so.1 with build-id 5dfba9be74e9275dc2b88197d5e4a7eb31caa30b
                                           Module libtirpc.so.3 with build-id 7face4554d6d0f0fbb939f024a850b7b37b4a32c
                                           Module libgfxdr.so.0 with build-id 62858b0a73c16da6933a848c1e333633c3013010
                                           Module libgfrpc.so.0 with build-id 50bc04de73cc7b46bb158a16de22bf0587d2b0f7
                                           Module libglusterfs.so.0 with build-id a3d7f3a8b7f8fd4122bb928970b76cccd64ec6c5
                                           Module glusterfsd with build-id 7117d73a79bc18905172667ba4c12692e2aa044b
                                           Stack trace of thread 1502:
                                           #0  0x00000000b6ed972c gf_io_run (libglusterfs.so.0 + 0x8972c)`

Additional info: I just upgraded glusterfs from 1:9.3-2 to 1:10.1-1, among other things on my system.

- The operating system / glusterfs version: glusterfs 1:10.1-1 os: 5.2.14-1-ARCH

Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration

[PisoMojado]

I rolled back only glusterfs to 1:9.3-2, and the system works again, that is glusterd runs successfully. I think, then, the issue is limited purely to the glusterfs upgrade. This is not a resolution, so I am not marking closed.

[mykaul]

Stack trace of thread 1502:

0 0x00000000b6ed972c gf_io_run (libglusterfs.so.0 + 0x8972c)`

@PisoMojado , can you get few more lines of that stack trace?

[PisoMojado]

unfortunately, that is the whole stack trace shown in the log

[mbirth]

I have this on a Raspberry Pi 2:

> sudo -H /usr/sbin/glusterd --debug -N
[2022-11-26 16:52:03.411014 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterd: Started running version [{arg=/usr/sbin/glusterd}, {version=10.1}, {cmdlinestr=/usr/sbin/glusterd --debug -N}]
[2022-11-26 16:52:03.411488 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 1795
[2022-11-26 16:52:03.411614 +0000] D [logging.c:1705:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
[2022-11-26 16:52:03.425049 +0000] D [MSGID: 0] [gf-io.c:513:gf_io_run] 0-io: Trying I/O engine 'legacy'
[2022-11-26 16:52:03.425430 +0000] D [MSGID: 0] [gf-io.c:517:gf_io_run] 0-io: I/O engine 'legacy' is ready
[2022-11-26 16:52:03.426937 +0000] D [logging.c:1675:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 3 extra log messages
[2022-11-26 16:52:03.427069 +0000] D [logging.c:1681:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 3 extra log messages
pending frames:
patchset: git://git.gluster.org/glusterfs.git
signal received: 4
time of crash:
2022-11-26 16:52:03 +0000
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 10.1
---------
fish: Job 1, 'sudo -H /usr/sbin/glusterd --de…' terminated by signal SIGILL (Illegal instruction)

This is with Ubuntu 22.04 and their latest glusterfs-server package. Same version runs fine on a Raspberry Pi 3 and 4 (both arm64 instead of armv7).

[xhernandez]

Latest versions of Gluster don't work on 32-bit architectures because they use 64-bit atomics, which are not supported and cause an "Illegal instruction" error.

[hostingnuggets]

Could I be affected by the same issue? I have opened the following issue recently https://github.com/gluster/glusterfs/issues/4177

[fkpwolf]

I have to compile it by myself with version 9.6 to let it run on my armhf device(Odroid HC1).

[gdistasi]

anything you can say on why I get error on my rasperry Pi 2 32 bits? Could I solve the issue if I recompile on the machine?