icolombi
commented
1 year ago Just happened again with the node cu-glstr-02-cl1. Same load (copying via Samba about 30 GB of data in 31k files).
Open icolombi opened 1 year ago
icolombi
commented
1 year ago Just happened again with the node cu-glstr-02-cl1. Same load (copying via Samba about 30 GB of data in 31k files).
rafikc30
commented
1 year ago @icolombi Do you have any core dump we can look at?
icolombi
commented
1 year ago How can I provide a core dump? Thanks
rafikc30
commented
1 year ago May be you can refer to this article based on Ubuntu 22.04
If you can find the core files, you can install the debug packages and attach the core files and get the backtrace using t a a bt, or the best would be sharing the corefiles and I will take a look at it.
icolombi
commented
1 year ago Thanks @rafikc30, I have the two dump files. Gzipped they are about 85 and 115 Mb, how can I share them with you?
rafikc30
commented
1 year ago @icolombi I think, The upload limit for attachments to a GitHub issue is currently 25 MB per file, you may want to consider using a file hosting service, such as Dropbox or Google Drive, and providing a link to the file in the GitHub issue.
icolombi
commented
1 year ago Thanks. Does the dump includes sensitive data?
rafikc30
commented
1 year ago A core file contains in-memory data while a process received the signal that caused the process to crash like SIGSEGV. Mostly, we are interested in the variable values, state of transport, ref count of variables etc. So In general it may contain file names, some metadata information, and I think it is also possible to have some content if a write or read is happening while the core is generated
rafikc30
commented
1 year ago @icolombi I will take a look
Expro
commented
1 year ago I'm experiencing same issue. I had cluster that was stable as a rock on 10.x, since update to 11.0 one one periodically crashes in similar way to one reported here.
madmax01
commented
1 year ago just as info..... on my side the last "stable" version is 10.1... every version after its just crashing after a period of time. something was essentially changed with 10.2
xImMoRtALitY99
commented
7 months ago @rafikc30 Did you find the root cause of this issue? We are facing those too, especially with larger directory trees having millions of files in it. There I can reproduce this issue all the time if you need additional coredump information. The OS is Debian 11 with GlusterFS Version 11. We didn't face those issues in GlusterFS Version 10.x or lower. I tried to compile GlusterFS with the "--enable-debug" parameter for more detailed coredump information too, however with that option enabled the brick unfortunately never crashed. Thanks in advance.
mohit84
commented
7 months ago @icolombi Can you please share "thread apply all bt full" output after attach a core with gdb.
LogicalNetworkingSolutions
commented
7 months ago Just wanted to add that I'm having the same issue ever since upgrading from Debian Bullseye to Bookworm (glusterfs-server 9.2 -> 10.3). 1 of 4 random gluster server processes seem to crash daily with similar to:
Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: pending frames: Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: patchset: git://git.gluster.org/glusterfs.git Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: signal received: 11 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: time of crash: Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: 2023-10-03 14:10:32 +0000 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: configuration details: Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: argp 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: backtrace 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: dlfcn 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: libpthread 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: llistxattr 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: setfsid 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: epoll.h 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: xattr.h 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: st_atim.tv_nsec 1 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: package-string: glusterfs 10.3 Oct 03 14:10:32 st04 srv-gluster-04-brick[2605345]: ---------
jeroenwichers
commented
5 months ago Yeah, we are experiencing this problem too. On a CentOS9 environment, with GlusterFS version 11.1. Brick log on my side is full with: [2023-12-21 14:14:59.494844 +0000] I [posix-entry-ops.c:382:posix_lookup] 0-avans-posix:
Maybe this can be helpful?
edrock200
commented
4 months ago Yeah, we are experiencing this problem too. On a CentOS9 environment, with GlusterFS version 11.1. Brick log on my side is full with: [2023-12-21 14:14:59.494844 +0000] I [posix-entry-ops.c:382:posix_lookup] 0-avans-posix: gfid:dd768023-41d8-4b18-a26c-782b418818a1/149c5ebf-688b-4543-bcac-6dfb1ab7ebbf: inode path not completely resolved. Asking for full path
Maybe this can be helpful?
On 11.1 and my brick logs are full of the same "inode path not completely resolved. Asking for full path" errors.
edrock200
commented
3 months ago Just to add to this, in digging a bit more, unless I'm misreading the logs, it appears the node the clients connect to to pull volume info appears to be "one version" old. By that I mean: Lets say I have 3 dispersed gluster nodes, each with a brick, and a 2+1 volume mounted to clients. Node 1 - brick 1 - port 1000 Node 2 - brick 2 - port 1001 Node 3 - brick 3 - port 1002
All clients mount by pointing to node 1.
At some point Node 2 crashes or I restart it. When it comes back up, the brick port changes to 2001. The clients still try to connect to 1001. If I kill the brick manually, restart glusterd, and it comes back online with 3001, the clients now try to connect to 2001. It's like it's advertising the port from the previous killed process for some reason.
Not sure if this matters but I do not have brick multiplexing enabled, nor shared storage, but I do have io_uring enabled.
nick-oconnor
commented
3 weeks ago @rafikc30 I'm running into this issue after upgrading from Ubuntu 22.04 with gluster 10.1 to Ubuntu 24.04 with gluster 11.1. I have multiple volumes, but the issue has only been triggered by a volume which backs a minio instance (lots of small file i/o):
Volume Name: minio
Type: Distribute
Volume ID: 1698d653-3c53-4955-b031-656951419885
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: nas-0:/pool-2/vol-0/minio/brick
Options Reconfigured:
diagnostics.brick-log-level: TRACE
performance.io-cache: on
performance.io-cache-size: 1GB
performance.quick-read-cache-timeout: 600
performance.parallel-readdir: on
performance.readdir-ahead: on
network.inode-lru-limit: 200000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
cluster.force-migration: off
performance.client-io-threads: on
cluster.readdir-optimize: on
diagnostics.client-log-level: ERROR
storage.fips-mode-rchecksum: on
transport.address-family: inetMy core dump is 1G due to cache settings and probably contains sensitive data, so I've only attached the brick backtrace and the last 10K lines of a trace-level brick log. Please let me know if there's anything else that would be helpful from the core dump.
nick-oconnor
commented
3 weeks ago I started looking through the 11 commits to inode.c since v10.1. I haven't found anything obvious that would cause inode to be null when passed to __inode_unref yet. Are there any relevant tests for this code?
mykaul
commented
3 weeks ago I started looking through the 11 commits to
inode.csince v10.1. I haven't found anything obvious that would causeinodeto be null when passed to __inode_unref yet. Are there any relevant tests for this code?
Could be https://github.com/gluster/glusterfs/commit/da2391dacd3483555e91a33ecdf89948be62b691
nick-oconnor
commented
3 weeks ago @mykaul yep, my core dump is exactly what's described in #4295 with ~5K recursive calls to inode_unref. I'll escalate this to the Ubuntu package maintainers and see if they'll patch it in.
Description of problem:
In a 3 replica cluster under heavy write load, one of the bricks become offline.
Mandatory info: - The output of the
gluster volume infocommand:- The output of the
gluster volume statuscommand:- The output of the
gluster volume healcommand:**- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/
On client side I have al lot of:
glusterd.log on the failed brick:
**- Is there any crash ? Provide the backtrace and coredump
On the failed brick log:
Additional info:
The restart of
glusterdprocess on the offline bricks recovered the situation. Now is healing the missing files.- The operating system / glusterfs version:
Ubuntu 22.04 LTS updated