Mr-Chengcheng
commented
1 year ago Today, using gdb for tracking, displayed in [ posix Readdirp Cbk ->posixpstat ]. The result returned -1, ia Type is IA_ INVAL, and found from one of the replicas that there is no link of directory under .glusterfs. (gdb) p hpath $6 = 0x7fde4c632540 "/gluster/brick-6/volume03/.glusterfs/58/d1/58d18981-257b-40d0-b1f4-c3d3d069f40b/." (gdb) l 5717 5718 strcpy(&hpath[len + 1], entry->d_name); 5719 5720 ret = posix_pstat(this, inode, gfid, hpath, &stbuf, _gf_false); 5721 5722 if (ret == -1) { 5723 if (inode) 5724 inode_unref(inode); 5725 continue; 5726 } (gdb) p ret $7 = -1 (gdb) p gfid $8 = '\000' <repeats 15 times> (gdb) p inode $9 = (inode_t *) 0x0 (gdb) p hpath $10 = 0x7fde4c632540 "/gluster/brick-6/volume03/.glusterfs/58/d1/58d18981-257b-40d0-b1f4-c3d3d069f40b/." (gdb) p stbuf $11 = {ia_flags = 0, ia_ino = 0, ia_dev = 0, ia_rdev = 0, ia_size = 0, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_mtime = 0, ia_ctime = 0, ia_btime = 0, ia_atime_nsec = 0, ia_mtime_nsec = 0, ia_ctime_nsec = 0, ia_btime_nsec = 0, ia_attributes = 0, ia_attributes_mask = 0, ia_gfid = '\000' <repeats 15 times>, ia_type = IA_INVAL, ia_prot = { suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = { read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}}
Description of problem: Recently, there have been some issues with the cluster I am using where directories cannot be deleted. Prompt 'Directory is not empty'. But the data exists on the brick and the extended attributes are normal. But it can be accessed from fuse according to the absolute path, as: [ deployer@ Alphanum sort] $ls - al Total 0 [ deployer@Alphanum sort] $ls - al package.json -Rw-r - r -1 root root 1599 May 31 16:02 package.json
I set the debug log to DEBUG and it displays: 15:19:52.171387] T [rpc-clnt.c:662:rpc_clnt_reply_init] 21-zjnbvol01cl01-client-5: received rpc message (RPC XID: 0x9154 Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 40) from rpc-transport (zjnbvol01cl01-client-5) [2023-06-01 15:19:52.171403] T [MSGID: 0] [client-rpc-fops_v2.c:2363:client4_0_readdirp_cbk] 21-stack-trace: stack-address: 0x7ff03c023cb8, zjnbvol01cl01-client-5 returned 3 [2023-06-01 15:19:52.171419] T [MSGID: 0] [afr-dir-read.c:233:afr_readdir_cbk] 21-stack-trace: stack-address: 0x7ff03c023cb8, zjnbvol01cl01-replicate-2 returned 3 [2023-06-01 15:19:52.171425] D [MSGID: 0] [dht-common.c:6845:dht_readdirp_cbk] 21-zjnbvol01cl01-dht: Processing entries from zjnbvol01cl01-replicate-2 [2023-06-01 15:19:52.171433] D [MSGID: 0] [dht-common.c:6852:dht_readdirp_cbk] 21-zjnbvol01cl01-dht: zjnbvol01cl01-replicate-2: entry = ., type = 4 [2023-06-01 15:19:52.171441] D [MSGID: 0] [dht-common.c:6860:dht_readdirp_cbk] 21-zjnbvol01cl01-dht: Invalid stat, ignoring entry . gfid 00000000-0000-0000-0000-000000000000 [Invalid argument] [2023-06-01 15:19:52.171451] D [MSGID: 0] [dht-common.c:6852:dht_readdirp_cbk] 21-zjnbvol01cl01-dht: zjnbvol01cl01-replicate-2: entry = .., type = 4 [2023-06-01 15:19:52.171466] D [MSGID: 0] [dht-common.c:6860:dht_readdirp_cbk] 21-zjnbvol01cl01-dht: Invalid stat, ignoring entry .. gfid 00000000-0000-0000-0000-000000000000 [Invalid argument] [2023-06-01 15:19:52.171473] D [MSGID: 0] [dht-common.c:6852:dht_readdirp_cbk] 21-zjnbvol01cl01-dht: zjnbvol01cl01-replicate-2: entry = license, type = 8 [2023-06-01 15:19:52.171479] D [MSGID: 0] [dht-common.c:6860:dht_readdirp_cbk] 21-zjnbvol01cl01-dht: Invalid stat, ignoring entry license gfid 00000000-0000-0000-0000-000000000000 [Invalid argument] [2023-06-01 15:19:52.171486] T [MSGID: 0] [dht-common.c:6763:dht_queue_readdirp] 21-stack-trace: stack-address: 0x7ff03c023cb8, winding from zjnbvol01cl01-dht to zjnbvol01cl01-replicate-3 [2023-06-01 15:19:52.171498] D [MSGID: 0] [afr-read-txn.c:468:afr_read_txn] 21-zjnbvol01cl01-replicate-3: f4e90d03-2ff0-4fc3-8e99-60fd76f82596: generation now vs cached: 2, 2 [2023-06-01 15:19:52.171506] T [MSGID: 0] [afr-dir-read.c:273:afr_readdir_wind] 21-stack-trace: stack-address: 0x7ff03c023cb8, winding from zjnbvol01cl01-replicate-3 to zjnbvol01cl01-client-7 [2023-06-01 15:19:52.171519] T [rpc-clnt.c:1444:rpc_clnt_record_build_header] 21-rpc-clnt: Request fraglen 268, payload: 180, rpc hdr: 88 [2023-06-01 15:19:52.171536] T [rpc-clnt.c:1731:rpc_clnt_submit] 21-rpc-clnt: submitted request (unique: 1502578, XID: 0x9141, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 40) to rpc-transport (zjnbvol01cl01-client-7) [2023-06-01 15:19:52.171547] T [socket.c:3007:socket_event_handler] 21-zjnbvol01cl01-client-5: (sock:13) socket_event_poll_in returned 0 [2023-06-01 15:19:52.171834] T [socket.c:2963:socket_event_handler] 21-zjnbvol01cl01-client-7: client (sock:23) in:1, out:0, err:0 [2023-06-01 15:19:52.171842] T [socket.c:2989:socket_event_handler] 21-zjnbvol01cl01-client-7: Client socket (23) is already connected [2023-06-01 15:19:52.171846] T [socket.c:581:socket_ssl_readv] 21-zjnbvol01cl01-client-7: ***** reading over non-SSL [2023-06-01 15:19:52.171854] T [socket.c:581:socket_ssl_readv] 21-zjnbvol01cl01-client-7: ***** reading over non-SSL
And it was found that the data was only read from one replica, while the state of the second replica was also healthy
Mandatory info: - The output of the
gluster volume infocommand: No data waiting for self repair - The output of thegluster volume statuscommand: All bricks are healthy - The output of thegluster volume healcommand: The self-healing process is normal**- Is there any crash ? Provide the backtrace and coredump no
- The operating system / glusterfs version: glusterfs 6.0
Parameter settings Options Reconfigured: cluster.choose-local: true cluster.lookup-unhashed: on performance.stat-prefetch: off features.cache-invalidation: on performance.cache-invalidation: true cluster.lookup-optimize: off performance.nl-cache: off diagnostics.brick-log-level: INFO performance.force-readdirp: false performance.quick-read: off dht.force-readdirp: off performance.write-behind: off diagnostics.client-log-level: INFO performance.readdir-ahead-pass-through: true performance.md-cache-pass-through: true cluster.heal-timeout: 600 cluster.data-self-heal: on cluster.metadata-self-heal: on cluster.entry-self-heal: on nfs.export-dir: /migrate_test() server.tcp-user-timeout: 21 network.frame-timeout: 300 performance.md-cache-timeout: 30 performance.write-behind-window-size: 200MB performance.cache-size: 32MB performance.normal-prio-threads: 8 cluster.full-lock: no cluster.favorite-child-policy: mtime cluster.consistent-metadata: on cluster.eager-lock: off performance.rda-cache-limit: 100MB performance.parallel-readdir: off cluster.server-quorum-type: none storage.reserve: 0 storage.build-pgfid: on performance.readdir-ahead: off performance.read-ahead: off diagnostics.stats-dump-interval: 60 client.bind-insecure: on performance.client-io-threads: off nfs.disable: on transport.address-family: inet diagnostics.latency-measurement: on diagnostics.count-fop-hits: on features.quota: on features.inode-quota: on features.quota-deem-statfs: on nfs.event-threads: 32 server.outstanding-rpc-limit: 256 nfs.trusted-sync: on disperse.eager-lock: off cluster.readdir-optimize: off server.event-threads: 8 client.event-threads: 8 network.inode-lru-limit: 1048576 network.ping-timeout: 8 performance.io-thread-count: 8 performance.nfs.write-behind-window-size: 1073741824 nfs.export-volumes: OFF diagnostics.fop-audit: on cluster.server-quorum-ratio: 51%
I have tried multiple parameter adjustments, but none of them were effective. Can you give me some advice? Thank you very much