k-van-man
commented
1 year ago Open k-van-man opened 1 year ago
k-van-man
commented
1 year ago 
xhernandez
commented
1 year ago Gluster crashed because of an "illegal instruction". If I'm not wrong, Odroid HC1 is a 32-bit platform, and Gluster doesn't support it.
Most probably the reason of the crash is an attempt to execute a 64-bit atomic instruction, which is not supported in a 32-bit platform.
k-van-man
commented
1 year ago Aha, yes the Odroid HC1 is old and 32 bit, and running Ubuntu 22.04 Is there another solution to host a distributed file system on multiple ARM 32-bit processor system like the Odroid HC1?
k-van-man
commented
1 year ago I'm not the only one with old 32-bits cpu boards see this link
als-git
commented
1 year ago TLDR: glusterfs doesn't support 32bit anymore.
xhernandez: The illegal instruction has nothing do to with your architecture. It is an intentional "can't fix this, crash now" triggered by userspace-rcu being asked to do the impossible (in this case, atomically update a 64bit value on a 32bit arch).
The version of userspace-rcu used by glusterfs 10.3 has this snippet of code:
unsigned long __uatomic_add_return(void *addr, unsigned long val,
int len)
{
switch (len) {
case 1:
{
unsigned char result = val;
__asm__ __volatile__(
"lock; xaddb %1, %0"
: "+m"(*__hp(addr)), "+q" (result)
:
: "memory");
return result + (unsigned char)val;
}
case 2:
{
unsigned short result = val;
__asm__ __volatile__(
"lock; xaddw %1, %0"
: "+m"(*__hp(addr)), "+r" (result)
:
: "memory");
return result + (unsigned short)val;
}
case 4:
{
unsigned int result = val;
__asm__ __volatile__(
"lock; xaddl %1, %0"
: "+m"(*__hp(addr)), "+r" (result)
:
: "memory");
return result + (unsigned int)val;
}
#if (CAA_BITS_PER_LONG == 64)
case 8:
{
unsigned long result = val;
__asm__ __volatile__(
"lock; xaddq %1, %0"
: "+m"(*__hp(addr)), "+r" (result)
:
: "memory");
return result + (unsigned long)val;
}
#endif
}
/*
* generate an illegal instruction. Cannot catch this with
* linker tricks when optimizations are disabled.
*/
__asm__ __volatile__("ud2");
return 0;
}
On a 32bit arch, the len=8 branch will be excluded by the preprocessor (because the condition is false) and so for 64bit values, it falls through straight to the explicit invocation of "ud2" (undefined), which cause the SIGILL. This is an intentional get-out-of-here crash, because there is nothing useful the code do here except exit in a noisy fashion (to attract attention to the problem).
Older version of glusterfs might still work (I had glusterfs 8.2 running on NetBSD/i386).
xhernandez
commented
1 year ago TLDR: glusterfs doesn't support 32bit anymore.
xhernandez: The illegal instruction has nothing do to with your architecture. It is an intentional "can't fix this, crash now" triggered by userspace-rcu being asked to do the impossible (in this case, atomically update a 64bit value on a 32bit arch).
That's exactly why the problem is caused by the architecture of the server. userspace-rcu just replaces an impossible to encode operation (an atomic 64-bit instruction) with an "ud2" instruction, but the reason is the lack of support for 64-bit atomic operations.
Though there are not many efforts to make sure Gluster works on 32-bit architectures, its possible that I'll fix this problem soon (though there's no guarantee that everything else will work).
davemuench
commented
1 year ago Subscribing as I have a fleet of 10 HC2's I'd love to keep up to date. I absolutely agree that 32 bit in this day and age is a bit silly, but at the same time I have yet to find a similar 64 bit alternative.
evindunn
commented
1 year ago Here's how I got gluster 6 installed on ubuntu 22.04 armhf. Warning: this is a hack & your mileage may vary.
add-apt-repository ppa:gluster/glusterfs-6Emits warning cause there's no gluster 6 release for jammy. Then update /etc/apt/sources.list.d/gluster-ubuntu-glusterfs-6-jammy.list to look like
deb https://ppa.launchpadcontent.net/gluster/glusterfs-6/ubuntu/ focal main
# deb-src https://ppa.launchpadcontent.net/gluster/glusterfs-6/ubuntu/ focal main
# For old dependencies
deb http://ports.ubuntu.com/ubuntu-ports focal main
deb http://ports.ubuntu.com/ubuntu-ports focal-security main/etc/apt/preferences.d/99-gluster
Package: *gluster* *libgf*
Pin: release o=LP-PPA-gluster-glusterfs-6
Pin-Priority: 999apt clean all && apt-get update && apt install glusterfs-server
In case this is of interest to the devs, I get a python warning during apt install -y glusterfs-server
Setting up libgfapi0:armhf (6.10-ubuntu1~focal1) ...
Setting up glusterfs-common (6.10-ubuntu1~focal1) ...
Adding group `gluster' (GID 122) ...
Done.
/usr/lib/arm-linux-gnueabihf/glusterfs/python/syncdaemon/syncdutils.py:705: SyntaxWarning: "is" with a literal. Did you mean "=="?
if dirpath is "/":
Setting up glusterfs-client (6.10-ubuntu1~focal1) ...
Setting up glusterfs-server (6.10-ubuntu1~focal1) ...But systemctl enable --now glusterd seems to be running fine
Description of problem: On a fresh install Ubuntu 22.04 on Odroid HC1 (ARM® big.LITTLE™ ): Failed to start GlusterFS, a clustered file-system server.
The exact command to reproduce the issue:
root@hc1:~# systemctl enable --now glusterdThe full output of the command that failed:
Expected results: Running gluster deamon
Mandatory info: - The output of the
gluster volume infocommand:Connection failed. Please check if gluster daemon is operational.- The output of thegluster volume statuscommand:Connection failed. Please check if gluster daemon is operational.- The output of thegluster volume healcommand:Connection failed. Please check if gluster daemon is operational.**- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/**- Is there any crash ? Provide the backtrace and coredump
Additional info:
- The operating system / glusterfs version:
Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration