Gluster Linked File Issues during Healing

Hi team,

my healings never finish

------------------------------

/304/disk.0.snap/0
/164/disk.0.snap/0
/165/disk.1
/169/disk.1
/169/disk.0.snap/0
/165/disk.0.snap/0
/166
/163
/304/disk.1
/166/disk.0.snap/0
/164
/304
/169
/158
/158/disk.1
/159/disk.2.snap/0
/165
/159/disk.0.snap/0
/159
/158/disk.0.snap/0
/166/disk.1
/164/disk.1
/159/disk.1
------------------------------------

we use opennebula kvm (qcow2) on zfs

disk.0 -> disk.0.snap/0     (thats an symlink)

it seems other data fine healed -------but such above structure its not healing

when going into glfs heal file its spamming this

[2024-03-18 05:03:05.173110 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2618:client4_0_lookup_cbk] 0-gv0-client-7: remote operation failed. [{path=<gfid:7f229755-1de3-48d9-ba76-46fefdc03fc9>}, {gfid=7f229755-1de3-48d9-ba76-46fefdc03fc9}, {errno=2}, {error=No such file or directory}]
[2024-03-18 05:03:05.174733 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2618:client4_0_lookup_cbk] 0-gv0-client-7: remote operation failed. [{path=<gfid:75d3a05a-6557-48e3-8496-cdb25ca914cd>}, {gfid=75d3a05a-6557-48e3-8496-cdb25ca914cd}, {errno=2}, {error=No such file or directory}]
[2024-03-18 05:03:05.176023 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2618:client4_0_lookup_cbk] 0-gv0-client-7: remote operation failed. [{path=<gfid:e44822be-9500-45be-be04-f2a538eeee47>}, {gfid=e44822be-9500-45be-be04-f2a538eeee47}, {errno=2}, {error=No such file or directory}]
[2024-03-18 05:03:05.177582 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2618:client4_0_lookup_cbk] 0-gv0-client-7: remote operation failed. [{path=<gfid:9ff3110f-bdc0-4f0a-bdec-9dc114ab0281>}, {gfid=9ff3110f-bdc0-4f0a-bdec-9dc114ab0281}, {errno=2}, {error=No such file or directory}]
-----------------------

not sure why blames "no such file or director". something not right.....i added ppa for gluster 10 and upgraded gluster on ubuntu 22.04 from 10.1 to 10.5.. but its same.....

Is this fixed with latest 11.x ?

Options Reconfigured:

storage.linux-io_uring: off
cluster.granular-entry-heal: enable
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: off
cluster.favorite-child-policy: mtime
diagnostics.brick-log-level: WARNING
diagnostics.client-log-level: WARNING
network.compression.compression-level: 0
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.md-cache-timeout: 600
cluster.quorum-type: fixed
cluster.quorum-count: 1
cluster.server-quorum-type: server
cluster.lookup-optimize: off
network.ping-timeout: 10
server.tcp-user-timeout: 10
server.keepalive-time: 10
server.keepalive-interval: 2
server.keepalive-count: 5
client.event-threads: 4
server.event-threads: 4
nfs.event-threads: 4
performance.read-ahead: off
performance.quick-read: off
performance.client-io-threads: on
performance.cache-invalidation: on
performance.low-prio-threads: 32
cluster.locking-scheme: granular
cluster.shd-wait-qlength: 10000
performance.enable-least-priority: no
cluster.shd-max-threads: 8
cluster.server-quorum-ratio: 1%

gluster / glusterfs

Gluster Linked File Issues during Healing #4318