icolombi
commented
1 year ago Just happened again with the node cu-glstr-02-cl1. Same load (copying via Samba about 30 GB of data in 31k files).
Open icolombi opened 1 year ago
icolombi
commented
1 year ago Just happened again with the node cu-glstr-02-cl1. Same load (copying via Samba about 30 GB of data in 31k files).
rafikc30
commented
1 year ago @icolombi Do you have any core dump we can look at?
icolombi
commented
1 year ago How can I provide a core dump? Thanks
rafikc30
commented
1 year ago May be you can refer to this article based on Ubuntu 22.04
If you can find the core files, you can install the debug packages and attach the core files and get the backtrace using t a a bt, or the best would be sharing the corefiles and I will take a look at it.
icolombi
commented
1 year ago Thanks @rafikc30, I have the two dump files. Gzipped they are about 85 and 115 Mb, how can I share them with you?
rafikc30
commented
1 year ago @icolombi I think, The upload limit for attachments to a GitHub issue is currently 25 MB per file, you may want to consider using a file hosting service, such as Dropbox or Google Drive, and providing a link to the file in the GitHub issue.
icolombi
commented
1 year ago Thanks. Does the dump includes sensitive data?
rafikc30
commented
1 year ago A core file contains in-memory data while a process received the signal that caused the process to crash like SIGSEGV. Mostly, we are interested in the variable values, state of transport, ref count of variables etc. So In general it may contain file names, some metadata information, and I think it is also possible to have some content if a write or read is happening while the core is generated
rafikc30
commented
1 year ago @icolombi I will take a look
Expro
commented
1 year ago I'm experiencing same issue. I had cluster that was stable as a rock on 10.x, since update to 11.0 one one periodically crashes in similar way to one reported here.
madmax01
commented
1 year ago just as info..... on my side the last "stable" version is 10.1... every version after its just crashing after a period of time. something was essentially changed with 10.2
xImMoRtALitY99
commented
1 year ago @rafikc30 Did you find the root cause of this issue? We are facing those too, especially with larger directory trees having millions of files in it. There I can reproduce this issue all the time if you need additional coredump information. The OS is Debian 11 with GlusterFS Version 11. We didn't face those issues in GlusterFS Version 10.x or lower. I tried to compile GlusterFS with the "--enable-debug" parameter for more detailed coredump information too, however with that option enabled the brick unfortunately never crashed. Thanks in advance.
mohit84
commented
1 year ago @icolombi Can you please share "thread apply all bt full" output after attach a core with gdb.
LogicalNetworkingSolutions
commented
1 year ago Just wanted to add that I'm having the same issue ever since upgrading from Debian Bullseye to Bookworm (glusterfs-server 9.2 -> 10.3). 1 of 4 random gluster server processes seem to crash daily with similar to:
Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: pending frames: Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: patchset: git://git.gluster.org/glusterfs.git Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: signal received: 11 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: time of crash: Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: 2023-10-03 14:10:32 +0000 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: configuration details: Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: argp 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: backtrace 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: dlfcn 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: libpthread 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: llistxattr 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: setfsid 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: epoll.h 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: xattr.h 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: st_atim.tv_nsec 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: package-string: glusterfs 10.3 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: ---------
jeroenwichers
commented
10 months ago Yeah, we are experiencing this problem too. On a CentOS9 environment, with GlusterFS version 11.1. Brick log on my side is full with: [2023-12-21 14:14:59.494844 +0000] I [posix-entry-ops.c:382:posix_lookup] 0-avans-posix:
Maybe this can be helpful?
edrock200
commented
10 months ago Yeah, we are experiencing this problem too. On a CentOS9 environment, with GlusterFS version 11.1. Brick log on my side is full with: [2023-12-21 14:14:59.494844 +0000] I [posix-entry-ops.c:382:posix_lookup] 0-avans-posix: gfid:dd768023-41d8-4b18-a26c-782b418818a1/149c5ebf-688b-4543-bcac-6dfb1ab7ebbf: inode path not completely resolved. Asking for full path
Maybe this can be helpful?
On 11.1 and my brick logs are full of the same "inode path not completely resolved. Asking for full path" errors.
edrock200
commented
9 months ago Just to add to this, in digging a bit more, unless I'm misreading the logs, it appears the node the clients connect to to pull volume info appears to be "one version" old. By that I mean: Lets say I have 3 dispersed gluster nodes, each with a brick, and a 2+1 volume mounted to clients. Node 1 - brick 1 - port 1000 Node 2 - brick 2 - port 1001 Node 3 - brick 3 - port 1002
All clients mount by pointing to node 1.
At some point Node 2 crashes or I restart it. When it comes back up, the brick port changes to 2001. The clients still try to connect to 1001. If I kill the brick manually, restart glusterd, and it comes back online with 3001, the clients now try to connect to 2001. It's like it's advertising the port from the previous killed process for some reason.
Not sure if this matters but I do not have brick multiplexing enabled, nor shared storage, but I do have io_uring enabled.
nick-oconnor
commented
6 months ago @rafikc30 I'm running into this issue after upgrading from Ubuntu 22.04 with gluster 10.1 to Ubuntu 24.04 with gluster 11.1. I have multiple volumes, but the issue has only been triggered by a volume which backs a minio instance (lots of small file i/o):
Volume Name: minio
Type: Distribute
Volume ID: 1698d653-3c53-4955-b031-656951419885
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: nas-0:/pool-2/vol-0/minio/brick
Options Reconfigured:
diagnostics.brick-log-level: TRACE
performance.io-cache: on
performance.io-cache-size: 1GB
performance.quick-read-cache-timeout: 600
performance.parallel-readdir: on
performance.readdir-ahead: on
network.inode-lru-limit: 200000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
cluster.force-migration: off
performance.client-io-threads: on
cluster.readdir-optimize: on
diagnostics.client-log-level: ERROR
storage.fips-mode-rchecksum: on
transport.address-family: inetMy core dump is 1G due to cache settings and probably contains sensitive data, so I've only attached the brick backtrace and the last 10K lines of a trace-level brick log. Please let me know if there's anything else that would be helpful from the core dump.
nick-oconnor
commented
6 months ago I started looking through the 11 commits to inode.c since v10.1. I haven't found anything obvious that would cause inode to be null when passed to __inode_unref yet. Are there any relevant tests for this code?
mykaul
commented
6 months ago I started looking through the 11 commits to
inode.csince v10.1. I haven't found anything obvious that would causeinodeto be null when passed to __inode_unref yet. Are there any relevant tests for this code?
Could be https://github.com/gluster/glusterfs/commit/da2391dacd3483555e91a33ecdf89948be62b691
nick-oconnor
commented
6 months ago @mykaul yep, my core dump is exactly what's described in #4295 with ~5K recursive calls to inode_unref. I'll escalate this to the Ubuntu package maintainers and see if they'll patch it in.
Trae32566
commented
5 months ago This also affects RHEL 9.4 and is a consistent issue for me; I've gotten to the point where I have to have a watchdog script in cron watching the process to make sure it doesn't die. Let me know if you need any additional troubleshooting info to fix this.
nick-oconnor
commented
5 months ago @Trae32566 that's unfortunate that this bug also affects RHEL. I don't think there's any troubleshooting left to do though as #4302 fixed issue. No release has the fix yet per https://github.com/gluster/glusterfs/issues/4295#issuecomment-2094665030. It's probably worth while adding a comment to that thread mentioning the impact to RHEL. Hopefully someone will cut a new release.
mykaul
commented
5 months ago @Trae32566 that's unfortunate that this bug also affects RHEL. I don't think there's any troubleshooting left to do though as #4302 fixed issue. No release has the fix yet per #4295 (comment). It's probably worth while adding a comment to that thread mentioning the impact to RHEL. Hopefully someone will cut a new release.
I don't think it's OS specific. It just needs a Glusterfs release. @gluster/gluster-maintainers can do it.
nick-oconnor
commented
5 months ago Yep, this is not OS-specific. The stack overflow is a stack overflow on any OS. However since there's no tagged release which contains the fix, every distro is gradually picking up bugged versions.
showinfo
commented
5 months ago I am using Gluster version 11.0 on Rocky Linux 8 and frequently experiencing sudden crashes of the same glusterfsd daemon. Currently, I don't have a solution, so I periodically check the daemon with cron and restart it.
When can I expect a fixed update version to be released?
nick-oconnor
commented
5 months ago @showinfo yeah I'm not sure what's going on. It seems like Gluster as a project is in a semi-maintained state after being dropped by Red Hat.
If you're feeling enterprising, can generate a patch and rebuild the rpm with something like this but for rpm (vs deb). The process for Rocky appears to be documented here.
edrock200
commented
5 months ago I am using Gluster version 11.0 on Rocky Linux 8 and frequently experiencing sudden crashes of the same glusterfsd daemon. Currently, I don't have a solution, so I periodically check the daemon with cron and restart it.
When can I expect a fixed update version to be released?
For reference 11.1 has been out for a while and did address a brick crash bug. Not sure if this is the same one you are experiencing though.
nick-oconnor
commented
5 months ago This is going to get interesting. Which PR was it? I'm referring to #4302 which is not part of 11.1.
intelliIT
commented
5 months ago same for 10.3 on Debian 12. had 3 random bricks goin offline in the last 4 days, following a restart of glusterd and extensive file-healing. always seems to follow big or rather many file operations.
edrock200
commented
5 months ago This is going to get interesting. Which PR was it? I'm referring to #4302 which is not part of 11.1.
Ahh my bad. Was referring to https://github.com/gluster/glusterfs/issues/4255
StefanBiermann
commented
4 months ago same for 10.3 on Debian 12. had 3 random bricks goin offline in the last 4 days, following a restart of glusterd and extensive file-healing. always seems to follow big or rather many file operations.
Same problem here, randomly crashing glusterd service. Fedora 39, GlusterFS 11.1
With big / many file operations at least one of three bricks crashes. Always the same - part of the LOG on glusterfs2.izn-ffm.intern:
[2024-07-03 00:08:26.554502 +0000] I [MSGID: 106143] [glusterd-pmap.c:353:pmap_port_remove] 0-pmap: removing brick (null) on port 52135 [2024-07-03 00:08:26.555741 +0000] I [MSGID: 106005] [glusterd-handler.c:6419:__glusterd_brick_rpc_notify] 0-management: Brick glusterfs2.izn-ffm.intern:/data/brick1/gv0 has disconnected from glusterd. [2024-07-03 00:08:26.556053 +0000] I [MSGID: 106143] [glusterd-pmap.c:353:pmap_port_remove] 0-pmap: removing brick /data/brick1/gv0 on port 52135
Besides that no errors in the logs. Easy config:
Volume Name: gv0 Type: Distribute Volume ID: 7242813e-e883-4ea0-9fe3-76f80c683639 Status: Started Snapshot Count: 0 Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: glusterfs1.izn-ffm.intern:/data/brick1/gv0 Brick2: glusterfs2.izn-ffm.intern:/data/brick1/gv0 Brick3: glusterfs3.izn-ffm.intern:/data/brick1/gv0 Options Reconfigured: diagnostics.brick-log-level: WARNING diagnostics.client-log-level: WARNING storage.fips-mode-rchecksum: on transport.address-family: inet cluster.daemon-log-level: WARNING
Restarting the daemon multiple times a day :(
Trae32566
commented
2 months ago Can someone please push a release to the repositories? Do I need to look at building it and hosting it? This is breaking the shit out of everything Fedora / Red Hat and I'm sick of it, to the point where I'm starting to look at replacements for GlusterFS in its entirety.
Trae32566
commented
2 months ago @StefanBiermann two options:
#!/usr/bin/env bash
brickHosts=(
'stor01a-rh9.int.trae32566.org'
'stor01b-rh9.int.trae32566.org'
'stor01c-rh9.int.trae32566.org'
)volumes=( 'jellyfin' )
for brickHost in "${brickHosts[@]}"; do for vol in "${volumes[@]}"; do volOnline=$(/usr/sbin/gluster volume status "${vol}" "${brickHost}:/mnt/bricks/${vol}/brick" detail | awk '/Online/ { print $3 }')
if [ "${volOnline}" != 'Y' ]; then
/usr/sbin/gluster volume start "${vol}" force
fi
donedone
2. If you're on Fedora, it's fairly easy to build using the instructions on their website on the branch `11dev`. I have had this branch built for a few days now and running on mine (RHEL 9) and no crashes so far, even with dual 40GbE and NVMe SSDs on both hosts.@StefanBiermann two options:
- You can use a script in cron to restart it automatically in the interrim, something like this, with brickHosts and volumes replaced with your brick hosts and volumes:
#!/usr/bin/env bash brickHosts=( 'stor01a-rh9.int.trae32566.org' 'stor01b-rh9.int.trae32566.org' 'stor01c-rh9.int.trae32566.org' ) volumes=( 'jellyfin' ) for brickHost in "${brickHosts[@]}"; do for vol in "${volumes[@]}"; do volOnline=$(/usr/sbin/gluster volume status "${vol}" "${brickHost}:/mnt/bricks/${vol}/brick" detail | awk '/Online/ { print $3 }') if [ "${volOnline}" != 'Y' ]; then /usr/sbin/gluster volume start "${vol}" force fi done done
- If you're on Fedora, it's fairly easy to build using the instructions on their website on the branch
11dev. I have had this branch built for a few days now and running on mine (RHEL 9) and no crashes so far, even with dual 40GbE and NVMe SSDs on both hosts.
I attempted to build 11dev from source on ubuntu jammy using the instructions on dev. Although I was able to install all dependencies, clone repo, configure, make, then make install, gluster wouldn't run due to missing symbol errors unfortunately on my test box. I tried both leaving the everything in configure as default and tried changing the prefix path to not use /usr/local, both attempts failed.
edrock200
commented
2 months ago @mohit84 is there anyone that can be nudged to push this patch to the repos in an updated version?
edrock200
commented
1 month ago Well this is disheartening....
xImMoRtALitY99
commented
1 month ago Well, this project is another great proof why you shouldn't trust anything related to Red Hat. Even though the devs care, Red Hat doesn't.
mykaul
commented
1 month ago Well, this project is another great proof why you shouldn't trust anything related to Red Hat. Even though the devs care, Red Hat doesn't.
You seem to have a gross misunderstanding of what open source is (vs enterprise / commercial offerings).
ronnyadsetts
commented
1 month ago Are there active non-Red Hat devs with sufficient access to the repository to merge fixes and make a release? I have no idea...
mykaul
commented
1 month ago Are there active non-Red Hat devs with sufficient access to the repository to merge fixes and make a release? I have no idea...
Not all owners (https://github.com/orgs/gluster/people?query=role%3Aowner ) are Red Hatters (or IBM today). You could ping them.
edrock200
commented
1 week ago So no one can push new binaries to the repos?
Description of problem:
In a 3 replica cluster under heavy write load, one of the bricks become offline.
Mandatory info: - The output of the
gluster volume infocommand:- The output of the
gluster volume statuscommand:- The output of the
gluster volume healcommand:**- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/
On client side I have al lot of:
glusterd.log on the failed brick:
**- Is there any crash ? Provide the backtrace and coredump
On the failed brick log:
Additional info:
The restart of
glusterdprocess on the offline bricks recovered the situation. Now is healing the missing files.- The operating system / glusterfs version:
Ubuntu 22.04 LTS updated