search

gluster / glusterfs

Gluster Filesystem : Build your distributed storage in minutes
https://www.gluster.org
GNU General Public License v2.0
4.74k stars 1.08k forks source link

GlusterFS rebalance after add-brick leaves 0 byte files with ---------T+ file permissions #4122

Open vinayak-gagrani opened 1 year ago

vinayak-gagrani commented 1 year ago

Description of problem: We had a glusterfs with 1 brick and mounted on several compute servers via network in a data centre. We wanted to expand volume so added 2 new bricks. We then ran rebalance with fix-layout option. Gluster was unmounted from all compute nodes before all this. Once rebalance was finished after 36 hours and we mounted filesystem back on clients, we started to see files with invalid permissions. These files are present even on the storage servers.

The exact command to reproduce the issue:

gluster volume rebalance distvol fix-layout start

The full output of the command that failed:

**Expected results:**
**Mandatory info:** **- The output of the `gluster volume info` command**: [root@ti-mum1-gluster3 titan_desk]# gluster volume info Volume Name: distvol Type: Distribute Volume ID: d7d45d0f-24a4-479e-a5bb-c473257932b0 Status: Started Snapshot Count: 0 Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 10.40.164.254:/bricks/dist_brick3/brick Brick2: 10.40.164.252:/bricks/dist_brick1/brick Brick3: 10.40.164.253:/bricks/dist_brick2/brick Options Reconfigured: performance.parallel-readdir: on server.outstanding-rpc-limit: 128 performance.io-thread-count: 64 server.event-threads: 24 client.event-threads: 32 storage.fips-mode-rchecksum: off transport.address-family: inet nfs.disable: on network.inode-lru-limit: 50000 performance.cache-invalidation: on features.cache-invalidation: on performance.qr-cache-timeout: 600 features.cache-invalidation-timeout: 600 performance.client-io-threads: on **- The output of the `gluster volume status` command**: [root@ti-mum1-gluster3 titan_desk]# gluster volume status Status of volume: distvol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.40.164.254:/bricks/dist_brick3/bri ck 49152 0 Y 1105284 Brick 10.40.164.252:/bricks/dist_brick1/bri ck 49152 0 Y 191556 Brick 10.40.164.253:/bricks/dist_brick2/bri ck 49152 0 Y 1178774 **- The output of the `gluster volume heal` command**: **- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/ [2023-04-05 08:41:37.618289 +0000] : peer detach 10.40.164.252 : SUCCESS [2023-04-05 08:41:47.574587 +0000] : v status : SUCCESS [2023-04-05 08:41:47.574936 +0000] : v status : SUCCESS [2023-04-21 13:20:50.602331 +0000] : volume status : SUCCESS [2023-04-21 13:20:50.604418 +0000] : volume status : SUCCESS [2023-04-22 01:22:26.290018 +0000] : peer probe 10.40.164.252 : FAILED : Probe returned with Transport endpoint is not connected [2023-04-22 01:23:35.915357 +0000] : peer probe 10.40.164.252 : SUCCESS [2023-04-22 01:23:39.731337 +0000] : peer probe 10.40.164.253 : SUCCESS [2023-04-22 01:25:10.669125 +0000] : volume add-brick distvol 10.40.164.252:/bricks/dist_brick1/brick : SUCCESS [2023-04-22 01:25:24.700315 +0000] : volume add-brick distvol 10.40.164.253:/bricks/dist_brick2/brick : SUCCESS [2023-04-22 01:26:23.265329 +0000] : volume status : SUCCESS [2023-04-22 01:26:23.267862 +0000] : volume status : SUCCESS [2023-04-22 01:27:23.448896 +0000] : volume rebalance distvol fix-layout start : SUCCESS [2023-04-22 01:28:05.765095 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 01:29:24.970526 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 03:27:33.921258 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 03:53:25.251355 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 03:56:46.375234 +0000] : volume rebalance distvol start : SUCCESS [2023-04-22 03:56:52.587564 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 03:57:43.181777 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 06:16:22.840626 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 06:16:56.708056 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 06:17:02.069625 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 08:21:40.458651 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 11:37:47.872759 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 11:50:45.340112 +0000] : volume rebalance distvol status : SUCCESS [2023-04-22 14:44:49.798081 +0000] : volume rebalance distvol status : SUCCESS [2023-04-23 05:38:51.685564 +0000] : volume rebalance distvol status : SUCCESS [2023-04-23 05:42:30.022227 +0000] : volume rebalance distvol status : SUCCESS [2023-04-23 05:46:48.245651 +0000] : volume rebalance distvol status : SUCCESS [2023-04-23 07:46:18.896031 +0000] : volume rebalance distvol status : SUCCESS [2023-04-23 09:21:27.181213 +0000] : volume rebalance distvol status : SUCCESS [2023-04-23 11:11:50.491972 +0000] : volume status : SUCCESS [2023-04-23 11:11:50.498981 +0000] : volume status : SUCCESS [2023-04-23 11:13:00.699491 +0000] : volume status distvol : SUCCESS [2023-04-23 11:13:30.137149 +0000] : volume stop distvol : SUCCESS [2023-04-23 11:14:02.389555 +0000] : volume status distvol : FAILED : Volume distvol is not started [2023-04-23 11:14:42.053229 +0000] : volume status distvol : FAILED : Volume distvol is not started [2023-04-23 11:14:55.656566 +0000] : volume start distvol : SUCCESS [2023-04-23 11:14:59.924393 +0000] : volume status distvol : SUCCESS [2023-04-23 11:15:59.008976 +0000] : volume rebalance distvol fix-layout start : SUCCESS [2023-04-23 11:16:12.965736 +0000] : volume rebalance distvol status : SUCCESS [2023-04-23 11:41:54.583077 +0000] : volume rebalance distvol status : SUCCESS (END) **- Is there any crash ? Provide the backtrace and coredump **Additional info:**
**- The operating system / glusterfs version**: [root@ti-mum1-gluster2 titan_desk]# gluster --version glusterfs 9.3 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. **Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration**
thenamehasbeentake commented 1 year ago

Can you provide log of fuse client and new bricks.

client log file location:

mountpoint : /mnt/test
log file: /var/log/glusterfs/mnt-test.log

bricks log file directory: /var/log/glusterfs/bricks/

Maybe you can check new bricks' filesystem, firewall...

RohitRanjan127 commented 1 year ago

client fuse log : [2023-04-23 10:03:16.310669 +0000] I [MSGID: 100030] [glusterfsd.c:2683:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=9.3}, {cmdlinestr=/usr/sbin/glusterfs --acl --resolve-gids --process-name fuse --volfile-server=10.40.164.254 --volfile-id=/distvol /titan_data}] [2023-04-23 10:03:16.312178 +0000] I [glusterfsd.c:2418:daemonize] 0-glusterfs: Pid of current running process is 41646 [2023-04-23 10:03:16.320511 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] [2023-04-23 10:03:16.320577 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] [2023-04-23 10:03:16.344654 +0000] I [glusterfsd-mgmt.c:2171:mgmt_getspec_cbk] 0-glusterfs: Received list of available volfile servers: 10.40.164.252:24007 10.40.164.253:24007 [2023-04-23 10:03:16.349974 +0000] I [io-stats.c:3708:ios_sample_buf_size_configure] 0-distvol: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0 [2023-04-23 10:03:16.350622 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}] [2023-04-23 10:03:16.350716 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}] [2023-04-23 10:03:16.350749 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=4}] [2023-04-23 10:03:16.351180 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=5}] [2023-04-23 10:03:16.351205 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=6}] [2023-04-23 10:03:16.351853 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=7}] [2023-04-23 10:03:16.351918 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=8}] [2023-04-23 10:03:16.351946 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=9}] [2023-04-23 10:03:16.351994 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=10}] [2023-04-23 10:03:16.352045 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=11}] [2023-04-23 10:03:16.352111 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=12}] [2023-04-23 10:03:16.352181 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=13}] [2023-04-23 10:03:16.352230 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=14}] [2023-04-23 10:03:16.352291 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=15}] [2023-04-23 10:03:16.352341 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=16}] [2023-04-23 10:03:16.352400 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=17}] [2023-04-23 10:03:16.352450 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=18}] [2023-04-23 10:03:16.353085 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=19}] [2023-04-23 10:03:16.353112 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=20}] [2023-04-23 10:03:16.353208 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=21}] [2023-04-23 10:03:16.353289 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=22}] [2023-04-23 10:03:16.353324 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=23}] [2023-04-23 10:03:16.353378 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=24}] [2023-04-23 10:03:16.353429 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=25}] [2023-04-23 10:03:16.353487 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=26}] [2023-04-23 10:03:16.353537 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=27}] [2023-04-23 10:03:16.354155 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=28}] [2023-04-23 10:03:16.354218 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=29}] [2023-04-23 10:03:16.354274 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=30}] [2023-04-23 10:03:16.354335 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=31}] [2023-04-23 10:03:16.354914 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-distvol-client-2: parent translators are ready, attempting connect on transport [] [2023-04-23 10:03:16.357579 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-distvol-client-3: parent translators are ready, attempting connect on transport [] [2023-04-23 10:03:16.357999 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 0-distvol-client-2: changing port to 49152 (from 0) [2023-04-23 10:03:16.358034 +0000] I [socket.c:849:__socket_shutdown] 0-distvol-client-2: intentional socket shutdown(12) [2023-04-23 10:03:16.360057 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-distvol-client-4: parent translators are ready, attempting connect on transport [] [2023-04-23 10:03:16.360606 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 0-distvol-client-3: changing port to 49152 (from 0) [2023-04-23 10:03:16.360634 +0000] I [socket.c:849:__socket_shutdown] 0-distvol-client-3: intentional socket shutdown(14) [2023-04-23 10:03:16.362741 +0000] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-distvol-client-2: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] [2023-04-23 10:03:16.365585 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 0-distvol-client-4: changing port to 49152 (from 0) [2023-04-23 10:03:16.365623 +0000] I [socket.c:849:__socket_shutdown] 0-distvol-client-4: intentional socket shutdown(13) [2023-04-23 10:03:16.365827 +0000] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-distvol-client-2: Connected, attached to remote volume [{conn-name=distvol-client-2}, {remote_subvol=/bricks/dist_brick3/brick}] [2023-04-23 10:03:16.368036 +0000] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-distvol-client-3: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] [2023-04-23 10:03:16.368447 +0000] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-distvol-client-3: Connected, attached to remote volume [{conn-name=distvol-client-3}, {remote_subvol=/bricks/dist_brick1/brick}] [2023-04-23 10:03:16.370429 +0000] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 0-distvol-client-4: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] [2023-04-23 10:03:16.370833 +0000] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 0-distvol-client-4: Connected, attached to remote volume [{conn-name=distvol-client-4}, {remote_subvol=/bricks/dist_brick2/brick}] [2023-04-23 10:03:16.372466 +0000] I [fuse-bridge.c:5296:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2023-04-23 10:03:16.372491 +0000] I [fuse-bridge.c:5924:fuse_graph_sync] 0-fuse: switched to graph 0 [2023-04-23 10:08:55.127396 +0000] I [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: e60253e6-5bb5-4be7-9f25-bc1f11a8907f, req(uid:5189,gid:4141,perm:1,ngrps:2), ctx(uid:20059,gid:4261,in-groups:0,perm:750,updated-fop:LOOKUP, acl:(tag:1,perm:7,id:4294967295)(tag:4,perm:5,id:4294967295)(tag:8,perm:5,id:4301,in-groups:0)(tag:16,perm:5,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] [2023-04-23 10:08:55.127443 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 46: LOOKUP() /sim_titan/titan_desk/python => -1 (Permission denied) [2023-04-23 10:08:55.127521 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 47: LOOKUP() /sim_titan/titan_desk/python => -1 (Permission denied) [2023-04-23 10:08:55.127592 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 48: LOOKUP() /sim_titan/titan_desk/research => -1 (Permission denied) [2023-04-23 10:08:55.127654 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 49: LOOKUP() /sim_titan/titan_desk/research => -1 (Permission denied) [2023-04-23 10:08:55.128778 +0000] I [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: cf826086-ce97-4371-88ba-ef96072ca86e, req(uid:5189,gid:4141,perm:1,ngrps:2), ctx(uid:20059,gid:4261,in-groups:0,perm:750,updated-fop:LOOKUP, acl:(tag:1,perm:7,id:4294967295)(tag:4,perm:5,id:4294967295)(tag:8,perm:5,id:4301,in-groups:0)(tag:16,perm:5,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] [2023-04-23 10:08:55.128799 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 51: LOOKUP() /sim_titan/tb_desk/tbscripts => -1 (Permission denied) The message "I [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: e60253e6-5bb5-4be7-9f25-bc1f11a8907f, req(uid:5189,gid:4141,perm:1,ngrps:2), ctx(uid:20059,gid:4261,in-groups:0,perm:750,updated-fop:LOOKUP, acl:(tag:1,perm:7,id:4294967295)(tag:4,perm:5,id:4294967295)(tag:8,perm:5,id:4301,in-groups:0)(tag:16,perm:5,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied]" repeated 3 times between [2023-04-23 10:08:55.127396 +0000] and [2023-04-23 10:08:55.127650 +0000] [2023-04-23 10:25:23.937948 +0000] I [fuse-bridge.c:6233:fuse_thread_proc] 0-fuse: initiating unmount of /titan_data [2023-04-23 10:25:23.938287 +0000] W [glusterfsd.c:1429:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7ff6297c0e25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55ef3202cc05] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef3202ca6b] ) 0-: received signum (15), shutting down

Brick log : [2023-04-23 00:58:50.112888 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/amahajan/other_users/sanand/results/config/NSE_JSWSTEEL/INDEX_SKIP/india.allday.index/sim/500us_Imp.1_1.LS12.AS15L5_25D_060F_25_Files_Z.0.0_40.0_QT/log [2023-04-23 00:58:50.112910 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/amahajan/other_users/sanand/results/config/NSE_JSWSTEEL/INDEX_SKIP/india.allday.index/sim/500us_Imp.1_1.LS12.AS15L5_25D_060F_25_Files_Z.0.0_40.0_QT/log [2023-04-23 01:01:34.480623 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/amahajan/configrepo/core.1416 [2023-04-23 01:01:34.480651 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/amahajan/configrepo/core.1416 [2023-04-23 01:04:41.858986 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.8621 [2023-04-23 01:04:41.859014 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.8621 [2023-04-23 01:04:41.859581 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.32214 [2023-04-23 01:04:41.859600 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.32214 [2023-04-23 01:04:41.872772 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.26897 [2023-04-23 01:04:41.872786 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.26897 [2023-04-23 01:04:41.873338 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.27003 [2023-04-23 01:04:41.873354 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.27003 [2023-04-23 01:04:41.873887 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.27313 [2023-04-23 01:04:41.873895 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.27313 [2023-04-23 01:04:41.875096 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.8988 [2023-04-23 01:04:41.875108 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/titan_ml/mlp/core.8988 [2023-04-23 01:07:33.097906 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/python2/python/pipeline/core.20157 [2023-04-23 01:07:33.097933 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/python2/python/pipeline/core.20157 [2023-04-23 01:07:33.098628 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/python2/python/pipeline/core.20481 [2023-04-23 01:07:33.098652 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/python2/python/pipeline/core.20481 [2023-04-23 01:07:49.564064 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/test/core.5149 [2023-04-23 01:07:49.564089 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/test/core.5149 [2023-04-23 01:07:49.674686 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/test/core.4677 [2023-04-23 01:07:49.674711 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/dmittal/titan_desk/test/core.4677 [2023-04-23 03:39:42.960054 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/sanand/results/config/NSE_JSWSTEEL/INDEX_SKIP/india.allday.index/sim/500us_Imp.1_1.LS12.AS15L5_25D_060F_25_Files_Z.0.0_40.0_QT/log [2023-04-23 03:39:42.960083 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/sanand/results/config/NSE_JSWSTEEL/INDEX_SKIP/india.allday.index/sim/500us_Imp.1_1.LS12.AS15L5_25D_060F_25_Files_Z.0.0_40.0_QT/log [2023-04-23 03:43:38.806950 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24891 [2023-04-23 03:43:38.806983 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24891 [2023-04-23 03:43:38.808121 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24925 [2023-04-23 03:43:38.808131 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24925 [2023-04-23 03:43:38.810501 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24933 [2023-04-23 03:43:38.810510 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24933 [2023-04-23 03:43:38.814467 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25005 [2023-04-23 03:43:38.814500 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25005 [2023-04-23 03:43:38.816014 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25133 [2023-04-23 03:43:38.816026 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25133 [2023-04-23 03:43:38.817499 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25165 [2023-04-23 03:43:38.817513 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25165 [2023-04-23 03:43:38.819582 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25169 [2023-04-23 03:43:38.819594 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25169 [2023-04-23 03:43:38.821606 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25188 [2023-04-23 03:43:38.821623 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25188 [2023-04-23 03:43:38.822915 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28439 [2023-04-23 03:43:38.822930 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28439 [2023-04-23 03:43:38.823878 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28444 [2023-04-23 03:43:38.823889 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28444 [2023-04-23 03:43:38.825164 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28521 [2023-04-23 03:43:38.825176 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28521 [2023-04-23 03:43:38.826147 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28524 [2023-04-23 03:43:38.826157 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28524 [2023-04-23 03:43:38.827551 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28523 [2023-04-23 03:43:38.827564 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28523 [2023-04-23 03:43:38.829171 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28522 [2023-04-23 03:43:38.829182 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28522 [2023-04-23 03:43:38.830415 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28520 [2023-04-23 03:43:38.830425 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28520 [2023-04-23 03:43:38.831872 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28553 [2023-04-23 03:43:38.831890 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28553 [2023-04-23 03:43:38.833444 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28554 [2023-04-23 03:43:38.833456 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28554 [2023-04-23 03:43:38.834839 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28556 [2023-04-23 03:43:38.834855 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28556 [2023-04-23 03:43:38.836431 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28557 [2023-04-23 03:43:38.836445 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28557 [2023-04-23 03:43:38.837192 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28555 [2023-04-23 03:43:38.837201 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28555 [2023-04-23 03:43:38.839066 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28579 [2023-04-23 03:43:38.839079 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28579 [2023-04-23 03:43:38.841474 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28576 [2023-04-23 03:43:38.841484 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28576 [2023-04-23 03:43:38.844304 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28578 [2023-04-23 03:43:38.844316 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28578 [2023-04-23 03:43:38.845363 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28658 [2023-04-23 03:43:38.845376 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28658 [2023-04-23 03:43:38.846323 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28655 [2023-04-23 03:43:38.846338 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28655 [2023-04-23 03:43:38.847445 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28657 [2023-04-23 03:43:38.847459 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28657 [2023-04-23 03:43:38.849450 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28683 [2023-04-23 03:43:38.849464 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28683 [2023-04-23 03:43:38.850597 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28686 [2023-04-23 03:43:38.850611 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28686 [2023-04-23 03:43:38.851581 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28687 [2023-04-23 03:43:38.851591 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28687 [2023-04-23 03:43:38.853023 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28685 [2023-04-23 03:43:38.853033 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28685 [2023-04-23 03:43:38.854525 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28707 [2023-04-23 03:43:38.854537 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28707 [2023-04-23 03:43:38.855251 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28710 [2023-04-23 03:43:38.855260 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28710 [2023-04-23 03:43:38.857134 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28711 [2023-04-23 03:43:38.857144 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28711 [2023-04-23 03:43:38.858574 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28797 [2023-04-23 03:43:38.858586 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28797 [2023-04-23 03:43:38.859339 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28799 [2023-04-23 03:43:38.859349 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28799 [2023-04-23 03:43:38.861770 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28794 [2023-04-23 03:43:38.861780 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28794 [2023-04-23 03:43:38.863627 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28795 [2023-04-23 03:43:38.863643 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28795 [2023-04-23 03:45:25.990701 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25044 [2023-04-23 03:45:25.990734 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25044 [2023-04-23 03:47:11.145357 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24806 [2023-04-23 03:47:11.145386 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24806 [2023-04-23 03:47:11.147830 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24878 [2023-04-23 03:47:11.147842 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24878 [2023-04-23 03:47:11.149954 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24905 [2023-04-23 03:47:11.149984 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24905 [2023-04-23 03:47:11.153039 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24912 [2023-04-23 03:47:11.153060 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.24912 [2023-04-23 03:47:11.155340 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25025 [2023-04-23 03:47:11.155354 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25025 [2023-04-23 03:47:11.158090 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25029 [2023-04-23 03:47:11.158102 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25029 [2023-04-23 03:47:11.438855 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25052 [2023-04-23 03:47:11.438874 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25052 [2023-04-23 03:47:11.441433 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25145 [2023-04-23 03:47:11.441445 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25145 [2023-04-23 03:47:11.443562 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25159 [2023-04-23 03:47:11.443573 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.25159 [2023-04-23 03:47:11.445701 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28440 [2023-04-23 03:47:11.445716 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28440 [2023-04-23 03:47:11.447781 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28438 [2023-04-23 03:47:11.447792 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28438 [2023-04-23 03:47:11.450205 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28437 [2023-04-23 03:47:11.450231 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28437 [2023-04-23 03:47:11.452394 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28441 [2023-04-23 03:47:11.452406 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28441 [2023-04-23 03:47:11.454381 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28525 [2023-04-23 03:47:11.454401 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28525 [2023-04-23 03:47:11.456550 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28558 [2023-04-23 03:47:11.456564 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28558 [2023-04-23 03:47:11.458986 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28580 [2023-04-23 03:47:11.459000 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28580 [2023-04-23 03:47:11.461105 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28581 [2023-04-23 03:47:11.461120 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28581 [2023-04-23 03:47:11.463233 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28577 [2023-04-23 03:47:11.463245 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28577 [2023-04-23 03:47:11.465560 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28654 [2023-04-23 03:47:11.465577 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28654 [2023-04-23 03:47:11.468050 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28656 [2023-04-23 03:47:11.468063 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28656 [2023-04-23 03:47:11.470211 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28653 [2023-04-23 03:47:11.470221 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28653 [2023-04-23 03:47:11.472491 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28684 [2023-04-23 03:47:11.472507 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28684 [2023-04-23 03:47:11.474674 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28682 [2023-04-23 03:47:11.474687 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28682 [2023-04-23 03:47:11.477205 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28709 [2023-04-23 03:47:11.477219 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28709 [2023-04-23 03:47:11.479308 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28712 [2023-04-23 03:47:11.479321 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28712 [2023-04-23 03:47:11.481468 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28708 [2023-04-23 03:47:11.481482 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28708 [2023-04-23 03:47:11.483622 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28796 [2023-04-23 03:47:11.483635 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28796 [2023-04-23 03:47:11.752092 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28835 [2023-04-23 03:47:11.752109 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28835 [2023-04-23 03:47:11.756414 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28836 [2023-04-23 03:47:11.756440 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28836 [2023-04-23 03:47:11.758837 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28833 [2023-04-23 03:47:11.758852 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28833 [2023-04-23 03:47:11.761115 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28837 [2023-04-23 03:47:11.761131 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28837 [2023-04-23 03:47:11.763148 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28834 [2023-04-23 03:47:11.763171 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28834 [2023-04-23 03:47:11.765181 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28832 [2023-04-23 03:47:11.765193 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.28832 [2023-04-23 03:47:11.767555 +0000] I [MSGID: 113030] [posix-entry-ops.c:1252:posix_unlink] 0-distvol-posix: open-fd-key-status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.23349 [2023-04-23 03:47:11.767570 +0000] I [MSGID: 113031] [posix-entry-ops.c:1143:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /bricks/dist_brick3/brick/TBStats/repos/tbscripts/exec/core.23349 exist and 2 new brick all use xfs filesystem and firewalld was stopped and disabled

thenamehasbeentake commented 1 year ago

It seems to be acl permits denied. You can try it again and see the logs on 3 bricks of the same time(exp: [2023-04-23 10:03:16.372466 +0000]). You might find something by comparing 3 bricks' log.

[2023-04-23 10:03:16.372466 +0000] I [fuse-bridge.c:5296:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2023-04-23 10:03:16.372491 +0000] I [fuse-bridge.c:5924:fuse_graph_sync] 0-fuse: switched to graph 0 [2023-04-23 10:08:55.127396 +0000] I [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: e60253e6-5bb5-4be7-9f25-bc1f11a8907f, req(uid:5189,gid:4141,perm:1,ngrps:2), ctx(uid:20059,gid:4261,in-groups:0,perm:750,updated-fop:LOOKUP, acl:(tag:1,perm:7,id:4294967295)(tag:4,perm:5,id:4294967295)(tag:8,perm:5,id:4301,in-groups:0)(tag:16,perm:5,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] [2023-04-23 10:08:55.127443 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 46: LOOKUP() /sim_titan/titan_desk/python => -1 (Permission denied) [2023-04-23 10:08:55.127521 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 47: LOOKUP() /sim_titan/titan_desk/python => -1 (Permission denied) [2023-04-23 10:08:55.127592 +0000] W [fuse-bridge.c:1076:fuse_entry_cbk] 0-glusterfs-fuse: 48: LOOKUP() /sim_titan/titan_desk/research => -1 (Permission denied)