search

gluster / gluster-geosync

Repository to implement Path based Geo-replication between two GlusterFS Volumes
Apache License 2.0
1 stars 0 forks source link

geo-rep stucks randomly #15

Open hunter86bg opened 2 years ago

hunter86bg commented 2 years ago 
OS (master): Ubuntu 18.04.5 LTS
OS(slave): Ubuntu 18.04.5 LTS

Session (hostnames replaced):

# gluster volume geo-replication status

MASTER NODE    MASTER VOL    MASTER BRICK                        SLAVE USER    SLAVE                  SLAVE NODE    STATUS     CRAWL STATUS       LAST_SYNCED                  
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
master2         gvol0         /nodirectwritedata/gluster/gvol0    root          ssh://slave3::gvol0    slave1        Active     Changelog Crawl    2021-12-01 11:29:04          
master3         gvol0         /nodirectwritedata/gluster/gvol0    root          ssh://slave3::gvol0    slave1        Passive    N/A                N/A                          
master1         gvol0         /nodirectwritedata/gluster/gvol0    root          ssh://slave3::gvol0    slave3        Passive    N/A                N/A

Time of issues (aprox):

# TZ='UTC' date --date='@1638306241'
Tue Nov 30 21:04:01 UTC 2021

# TZ='UTC' date --date='@1638386942'
Wed Dec  1 19:29:02 UTC 2021

Slave Log:

root@slave1:/var/log/glusterfs/geo-replication-slaves/gvol0_slave3_gvol0# TZ=UTC stat gsyncd.log
  File: gsyncd.log
  Size: 11083       Blocks: 24         IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 1446089     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-12-02 21:15:25.087806878 +0000
Modify: 2021-11-30 21:05:25.523976697 +0000
Change: 2021-11-30 21:05:25.523976697 +0000
 Birth: -

root@slave1:/var/log/glusterfs/geo-replication-slaves/gvol0_slave3_gvol0#  tail -n 5 gsyncd.log
[2021-11-02 08:28:41.529009] I [resource(slave master2/nodirectwritedata/gluster/gvol0):1166:service_loop] GLUSTER: slave listening
[2021-11-30 21:05:12.855912] I [repce(slave master3/nodirectwritedata/gluster/gvol0):96:service_loop] RepceServer: terminating on reaching EOF.
[2021-11-30 21:05:24.453134] I [resource(slave master3/nodirectwritedata/gluster/gvol0):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-11-30 21:05:25.527423] I [resource(slave master3/nodirectwritedata/gluster/gvol0):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.0737}]
[2021-11-30 21:05:25.528730] I [resource(slave master3/nodirectwritedata/gluster/gvol0):1166:service_loop] GLUSTER: slave listening

Master log:

[2021-12-01 19:25:04.348684] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16036 done, took 5.008963 seconds
[2021-12-01 19:25:05.230742] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140055754548992:1638386705.2305565 keep_alive({'version': (1, 0), 'uuid': '752ceb0d-77c2-42c5-9ba1-be43b34af460', 'retval': 0, 'volume_mark': (1621660285, 623270), 'timeout': 1638386825},) ...
[2021-12-01 19:25:05.250773] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140055754548992:1638386705.2305565 keep_alive -> 42370
[2021-12-01 19:25:09.359050] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16037 done, took 5.009990 seconds
[2021-12-01 19:25:14.370048] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16038 done, took 5.010609 seconds
[2021-12-01 19:25:19.380563] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16039 done, took 5.010139 seconds
[2021-12-01 19:25:24.388670] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16040 done, took 5.007689 seconds
[2021-12-01 19:25:29.399303] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16041 done, took 5.010244 seconds
[2021-12-01 19:25:34.409488] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16042 done, took 5.009907 seconds
[2021-12-01 19:25:39.419664] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16043 done, took 5.009842 seconds
[2021-12-01 19:25:44.430522] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16044 done, took 5.010487 seconds
[2021-12-01 19:25:44.430801] D [master(worker /nodirectwritedata/gluster/gvol0):561:crawlwrap] _GMaster: 12 crawls, 0 turns
[2021-12-01 19:25:48.589899] D [gsyncd(config-get):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:25:48.781122] D [gsyncd(status):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:25:49.440886] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16045 done, took 5.010044 seconds
[2021-12-01 19:25:49.899182] D [gsyncd(config-get):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:25:50.85084] D [gsyncd(status):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:25:51.340667] D [gsyncd(config-get):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:25:51.529013] D [gsyncd(status):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:25:54.448703] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16046 done, took 5.007515 seconds
[2021-12-01 19:25:59.459746] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16047 done, took 5.010657 seconds
[2021-12-01 19:26:04.468750] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16048 done, took 5.008729 seconds
[2021-12-01 19:26:05.311197] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140055754548992:1638386765.311053 keep_alive({'version': (1, 0), 'uuid': '752ceb0d-77c2-42c5-9ba1-be43b34af460', 'retval': 0, 'volume_mark': (1621660285, 623270), 'timeout': 1638386885},) ...
[2021-12-01 19:26:05.330971] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140055754548992:1638386765.311053 keep_alive -> 42371
[2021-12-01 19:26:09.479032] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16049 done, took 5.009957 seconds
[2021-12-01 19:26:14.489477] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16050 done, took 5.010071 seconds
[2021-12-01 19:26:19.499864] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16051 done, took 5.010052 seconds
[2021-12-01 19:26:24.510322] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16052 done, took 5.010114 seconds
[2021-12-01 19:26:29.521150] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16053 done, took 5.010426 seconds
[2021-12-01 19:26:34.531521] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16054 done, took 5.010051 seconds
[2021-12-01 19:26:39.541913] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16055 done, took 5.010068 seconds
[2021-12-01 19:26:44.555528] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16056 done, took 5.013210 seconds
[2021-12-01 19:26:44.555827] D [master(worker /nodirectwritedata/gluster/gvol0):561:crawlwrap] _GMaster: 12 crawls, 0 turns
[2021-12-01 19:26:49.566209] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16057 done, took 5.010321 seconds
[2021-12-01 19:26:54.579013] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16058 done, took 5.012434 seconds
[2021-12-01 19:26:59.592371] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16059 done, took 5.012943 seconds
[2021-12-01 19:27:04.605282] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16060 done, took 5.012503 seconds
[2021-12-01 19:27:05.391371] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140055754548992:1638386825.3912401 keep_alive({'version': (1, 0), 'uuid': '752ceb0d-77c2-42c5-9ba1-be43b34af460', 'retval': 0, 'volume_mark': (1621660285, 623270), 'timeout': 1638386945},) ...
[2021-12-01 19:27:05.411334] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140055754548992:1638386825.3912401 keep_alive -> 42372
[2021-12-01 19:27:09.615674] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16061 done, took 5.010036 seconds
[2021-12-01 19:27:14.629079] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16062 done, took 5.013035 seconds
[2021-12-01 19:27:19.642319] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16063 done, took 5.012856 seconds
[2021-12-01 19:27:24.655154] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16064 done, took 5.012458 seconds
[2021-12-01 19:27:29.664634] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16065 done, took 5.009085 seconds
[2021-12-01 19:27:34.676670] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16066 done, took 5.011742 seconds
[2021-12-01 19:27:39.688647] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16067 done, took 5.011621 seconds
[2021-12-01 19:27:44.701628] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16068 done, took 5.012573 seconds
[2021-12-01 19:27:44.701946] D [master(worker /nodirectwritedata/gluster/gvol0):561:crawlwrap] _GMaster: 12 crawls, 0 turns
[2021-12-01 19:27:49.712688] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16069 done, took 5.010685 seconds
[2021-12-01 19:27:54.725400] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16070 done, took 5.012349 seconds
[2021-12-01 19:27:59.738638] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16071 done, took 5.012847 seconds
[2021-12-01 19:28:04.751384] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16072 done, took 5.012387 seconds
[2021-12-01 19:28:05.471802] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140055754548992:1638386885.471657 keep_alive({'version': (1, 0), 'uuid': '752ceb0d-77c2-42c5-9ba1-be43b34af460', 'retval': 0, 'volume_mark': (1621660285, 623270), 'timeout': 1638387005},) ...
[2021-12-01 19:28:05.491692] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140055754548992:1638386885.471657 keep_alive -> 42373
[2021-12-01 19:28:09.764153] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16073 done, took 5.012397 seconds
[2021-12-01 19:28:14.777562] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16074 done, took 5.012975 seconds
[2021-12-01 19:28:19.791133] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16075 done, took 5.013165 seconds
[2021-12-01 19:28:24.804023] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16076 done, took 5.012506 seconds
[2021-12-01 19:28:29.817540] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16077 done, took 5.013085 seconds
[2021-12-01 19:28:34.828704] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16078 done, took 5.010765 seconds
[2021-12-01 19:28:39.841737] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16079 done, took 5.012664 seconds
[2021-12-01 19:28:44.852559] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16080 done, took 5.010371 seconds
[2021-12-01 19:28:44.852894] D [master(worker /nodirectwritedata/gluster/gvol0):561:crawlwrap] _GMaster: 12 crawls, 0 turns
[2021-12-01 19:28:49.863741] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16081 done, took 5.010792 seconds
[2021-12-01 19:28:54.876730] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16082 done, took 5.012628 seconds
[2021-12-01 19:28:54.885300] I [master(worker /nodirectwritedata/gluster/gvol0):1508:crawl] _GMaster: slave's time [{stime=(1638306276, 0)}]
[2021-12-01 19:28:54.886124] D [master(worker /nodirectwritedata/gluster/gvol0):1492:changelogs_batch_process] _GMaster: processing changes [{batch=['/var/lib/misc/gluster/gsyncd/gvol0_slave3_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1638386931']}]
[2021-12-01 19:28:54.886371] D [master(worker /nodirectwritedata/gluster/gvol0):1327:process] _GMaster: processing change [{changelog=/var/lib/misc/gluster/gsyncd/gvol0_slave3_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1638386931}]
[2021-12-01 19:28:54.891685] D [master(worker /nodirectwritedata/gluster/gvol0):1137:process_change] _GMaster: Ignoring entry, purged in the interim [{file=.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/.lock}, {gfid=8c77afee-ae50-42b5-9017-50f5070b6b9f}]
[2021-12-01 19:28:54.899050] D [master(worker /nodirectwritedata/gluster/gvol0):1206:process_change] _GMaster: entries: [{'op': 'UNLINK', 'skip_entry': False, 'gfid': '53aadcfe-c371-420d-b9a1-6013a82c42c3', 'entry': '.gfid/6f682112-50bb-453a-96da-bd86030904be/.lock-19804511'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '8c77afee-ae50-42b5-9017-50f5070b6b9f', 'entry': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/.lock-97530168', 'mode': 33152, 'uid': 108, 'gid': 117}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': '8c77afee-ae50-42b5-9017-50f5070b6b9f', 'entry': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/.lock-97530168'}, {'op': 'RENAME', 'skip_entry': False, 'gfid': '82827319-7fe4-477c-b6cc-e1a05188bb55', 'entry': '.gfid/6f682112-50bb-453a-96da-bd86030904be/fax_1638386914_16383868414176717_4119_8317551702__.pdf', 'entry1': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/fax_1638386914_16383868414176717_4119_8317551702__.pdf', 'stat': {'uid': 108, 'gid': 117, 'mode': 33204, 'atime': 1638386921.0243804, 'mtime': 1638386921.0243804}, 'link': None}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': '8c77afee-ae50-42b5-9017-50f5070b6b9f', 'entry': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/.lock'}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': 'bc9f58a0-d2d4-4fdf-b5e7-5b4c48a9c178', 'entry': '.gfid/6f682112-50bb-453a-96da-bd86030904be/.lock'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '0ef2f45f-b433-45bc-acd0-60218871be26', 'entry': '.gfid/c7c79516-44a0-4dcb-a599-6085f21a9adf/voicemail_1638386929_1638386915184642_40582.wav', 'mode': 33188, 'uid': 108, 'gid': 117}]
[2021-12-01 19:28:54.909793] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140056992864064:1638386934.9096642 entry_ops([{'op': 'UNLINK', 'skip_entry': False, 'gfid': '53aadcfe-c371-420d-b9a1-6013a82c42c3', 'entry': '.gfid/6f682112-50bb-453a-96da-bd86030904be/.lock-19804511'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '8c77afee-ae50-42b5-9017-50f5070b6b9f', 'entry': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/.lock-97530168', 'mode': 33152, 'uid': 108, 'gid': 117}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': '8c77afee-ae50-42b5-9017-50f5070b6b9f', 'entry': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/.lock-97530168'}, {'op': 'RENAME', 'skip_entry': False, 'gfid': '82827319-7fe4-477c-b6cc-e1a05188bb55', 'entry': '.gfid/6f682112-50bb-453a-96da-bd86030904be/fax_1638386914_16383868414176717_4119_8317551702__.pdf', 'entry1': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/fax_1638386914_16383868414176717_4119_8317551702__.pdf', 'stat': {'uid': 108, 'gid': 117, 'mode': 33204, 'atime': 1638386921.0243804, 'mtime': 1638386921.0243804}, 'link': None}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': '8c77afee-ae50-42b5-9017-50f5070b6b9f', 'entry': '.gfid/e06b89fc-0432-4594-9705-35c02d8ea856/.lock'}, {'op': 'UNLINK', 'skip_entry': False, 'gfid': 'bc9f58a0-d2d4-4fdf-b5e7-5b4c48a9c178', 'entry': '.gfid/6f682112-50bb-453a-96da-bd86030904be/.lock'}, {'op': 'CREATE', 'skip_entry': False, 'gfid': '0ef2f45f-b433-45bc-acd0-60218871be26', 'entry': '.gfid/c7c79516-44a0-4dcb-a599-6085f21a9adf/voicemail_1638386929_1638386915184642_40582.wav', 'mode': 33188, 'uid': 108, 'gid': 117}],) ...
[2021-12-01 19:28:54.967882] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140056992864064:1638386934.9096642 entry_ops -> []
[2021-12-01 19:28:54.997416] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140056992864064:1638386934.9973304 meta_ops([{'op': 'META', 'skip_entry': False, 'go': '.gfid/82827319-7fe4-477c-b6cc-e1a05188bb55', 'stat': {'uid': 108, 'gid': 117, 'mode': 33204, 'atime': 1638386921.0243804, 'mtime': 1638386921.0243804}}],) ...
[2021-12-01 19:28:55.23329] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140056992864064:1638386934.9973304 meta_ops -> []
[2021-12-01 19:28:55.70056] D [master(worker /nodirectwritedata/gluster/gvol0):316:a_syncdata] _GMaster: files [{files={'.gfid/94949690-4829-46d6-8feb-229525fa1a3a', '.gfid/0ef2f45f-b433-45bc-acd0-60218871be26', '.gfid/1f003902-e95c-482f-8828-187ba8a1a88d', '.gfid/40dc07f6-70b1-404e-9620-85e260d3cc34', '.gfid/c4e0454e-05fd-4b7e-87c6-a2e7d7fcbe31', '.gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2', '.gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586', '.gfid/06df7912-0af2-4539-9ef3-21da4cbb7715', '.gfid/8f508423-37cf-4095-9ffd-b7bb932cb022', '.gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d', '.gfid/aa58c901-90fe-4d9a-9f4d-0f1f11293048', '.gfid/899bef1e-4eef-4b44-84fc-491c5ff00d9a', '.gfid/85fb1f7d-ffca-49bc-b522-8e1fe371678d', '.gfid/82827319-7fe4-477c-b6cc-e1a05188bb55', '.gfid/46253285-a7ba-437f-a27c-ad281a39eda2'}}]
[2021-12-01 19:28:55.70288] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/94949690-4829-46d6-8feb-229525fa1a3a}]
[2021-12-01 19:28:55.70466] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/0ef2f45f-b433-45bc-acd0-60218871be26}]
[2021-12-01 19:28:55.70605] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/1f003902-e95c-482f-8828-187ba8a1a88d}]
[2021-12-01 19:28:55.70729] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/40dc07f6-70b1-404e-9620-85e260d3cc34}]
[2021-12-01 19:28:55.70865] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/c4e0454e-05fd-4b7e-87c6-a2e7d7fcbe31}]
[2021-12-01 19:28:55.72399] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2}]
[2021-12-01 19:28:55.72588] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586}]
[2021-12-01 19:28:55.72734] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/06df7912-0af2-4539-9ef3-21da4cbb7715}]
[2021-12-01 19:28:55.72874] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/8f508423-37cf-4095-9ffd-b7bb932cb022}]
[2021-12-01 19:28:55.72996] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d}]
[2021-12-01 19:28:55.73108] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/aa58c901-90fe-4d9a-9f4d-0f1f11293048}]
[2021-12-01 19:28:55.73219] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/899bef1e-4eef-4b44-84fc-491c5ff00d9a}]
[2021-12-01 19:28:55.73330] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/85fb1f7d-ffca-49bc-b522-8e1fe371678d}]
[2021-12-01 19:28:55.73440] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/82827319-7fe4-477c-b6cc-e1a05188bb55}]
[2021-12-01 19:28:55.73549] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/46253285-a7ba-437f-a27c-ad281a39eda2}]
[2021-12-01 19:28:55.114331] D [resource(worker /nodirectwritedata/gluster/gvol0):1442:rsync] SSH: files: .gfid/94949690-4829-46d6-8feb-229525fa1a3a, .gfid/0ef2f45f-b433-45bc-acd0-60218871be26, .gfid/1f003902-e95c-482f-8828-187ba8a1a88d, .gfid/40dc07f6-70b1-404e-9620-85e260d3cc34, .gfid/c4e0454e-05fd-4b7e-87c6-a2e7d7fcbe31, .gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2, .gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586, .gfid/06df7912-0af2-4539-9ef3-21da4cbb7715, .gfid/8f508423-37cf-4095-9ffd-b7bb932cb022, .gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d, .gfid/aa58c901-90fe-4d9a-9f4d-0f1f11293048, .gfid/899bef1e-4eef-4b44-84fc-491c5ff00d9a, .gfid/85fb1f7d-ffca-49bc-b522-8e1fe371678d, .gfid/82827319-7fe4-477c-b6cc-e1a05188bb55, .gfid/46253285-a7ba-437f-a27c-ad281a39eda2
[2021-12-01 19:28:55.476911] I [master(worker /nodirectwritedata/gluster/gvol0):1996:syncjob] Syncer: Sync Time Taken [{job=3}, {num_files=15}, {return_code=0}, {duration=0.3623}]
[2021-12-01 19:28:55.477926] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/94949690-4829-46d6-8feb-229525fa1a3a}]
[2021-12-01 19:28:55.478143] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/0ef2f45f-b433-45bc-acd0-60218871be26}]
[2021-12-01 19:28:55.478262] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/1f003902-e95c-482f-8828-187ba8a1a88d}]
[2021-12-01 19:28:55.478371] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/40dc07f6-70b1-404e-9620-85e260d3cc34}]
[2021-12-01 19:28:55.478463] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/c4e0454e-05fd-4b7e-87c6-a2e7d7fcbe31}]
[2021-12-01 19:28:55.478562] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2}]
[2021-12-01 19:28:55.478665] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586}]
[2021-12-01 19:28:55.478757] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/06df7912-0af2-4539-9ef3-21da4cbb7715}]
[2021-12-01 19:28:55.478844] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/8f508423-37cf-4095-9ffd-b7bb932cb022}]
[2021-12-01 19:28:55.478930] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d}]
[2021-12-01 19:28:55.479017] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/aa58c901-90fe-4d9a-9f4d-0f1f11293048}]
[2021-12-01 19:28:55.479111] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/899bef1e-4eef-4b44-84fc-491c5ff00d9a}]
[2021-12-01 19:28:55.479196] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/85fb1f7d-ffca-49bc-b522-8e1fe371678d}]
[2021-12-01 19:28:55.479279] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/82827319-7fe4-477c-b6cc-e1a05188bb55}]
[2021-12-01 19:28:55.479360] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/46253285-a7ba-437f-a27c-ad281a39eda2}]
[2021-12-01 19:28:55.492130] I [master(worker /nodirectwritedata/gluster/gvol0):1422:process] _GMaster: Entry Time Taken [{UNL=4}, {RMD=0}, {CRE=2}, {MKN=0}, {MKD=0}, {REN=1}, {LIN=1}, {SYM=0}, {duration=0.0875}]
[2021-12-01 19:28:55.492266] I [master(worker /nodirectwritedata/gluster/gvol0):1432:process] _GMaster: Data/Metadata Time Taken [{SETA=1}, {meta_duration=0.0820}, {SETX=0}, {XATT=0}, {DATA=15}, {data_duration=0.4223}]
[2021-12-01 19:28:55.492656] I [master(worker /nodirectwritedata/gluster/gvol0):1442:process] _GMaster: Batch Completed [{mode=live_changelog}, {duration=0.6060}, {changelog_start=1638386931}, {changelog_end=1638386931}, {num_changelogs=1}, {stime=(1638386930, 0)}, {entry_stime=(1638386930, 0)}]
[2021-12-01 19:29:00.497995] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16083 done, took 5.620840 seconds
[2021-12-01 19:29:05.509290] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16084 done, took 5.010901 seconds
[2021-12-01 19:29:05.514579] I [master(worker /nodirectwritedata/gluster/gvol0):1508:crawl] _GMaster: slave's time [{stime=(1638386930, 0)}]
[2021-12-01 19:29:05.515199] D [master(worker /nodirectwritedata/gluster/gvol0):1492:changelogs_batch_process] _GMaster: processing changes [{batch=['/var/lib/misc/gluster/gsyncd/gvol0_slave3_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1638386945']}]
[2021-12-01 19:29:05.515386] D [master(worker /nodirectwritedata/gluster/gvol0):1327:process] _GMaster: processing change [{changelog=/var/lib/misc/gluster/gsyncd/gvol0_slave3_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1638386945}]
[2021-12-01 19:29:05.517469] D [master(worker /nodirectwritedata/gluster/gvol0):1206:process_change] _GMaster: entries: [{'op': 'UNLINK', 'skip_entry': False, 'gfid': '0ef2f45f-b433-45bc-acd0-60218871be26', 'entry': '.gfid/c7c79516-44a0-4dcb-a599-6085f21a9adf/voicemail_1638386929_1638386915184642_40582.wav'}]
[2021-12-01 19:29:05.535880] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140056992864064:1638386945.535773 entry_ops([{'op': 'UNLINK', 'skip_entry': False, 'gfid': '0ef2f45f-b433-45bc-acd0-60218871be26', 'entry': '.gfid/c7c79516-44a0-4dcb-a599-6085f21a9adf/voicemail_1638386929_1638386915184642_40582.wav'}],) ...
[2021-12-01 19:29:05.552209] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140055754548992:1638386945.5520768 keep_alive({'version': (1, 0), 'uuid': '752ceb0d-77c2-42c5-9ba1-be43b34af460', 'retval': 0, 'volume_mark': (1621660285, 623270), 'timeout': 1638387065},) ...
[2021-12-01 19:29:05.561660] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140056992864064:1638386945.535773 entry_ops -> []
[2021-12-01 19:29:05.571693] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140055754548992:1638386945.5520768 keep_alive -> 42374
[2021-12-01 19:29:05.581405] D [master(worker /nodirectwritedata/gluster/gvol0):316:a_syncdata] _GMaster: files [{files={'.gfid/94949690-4829-46d6-8feb-229525fa1a3a', '.gfid/6ef035fe-99dc-42e1-94e8-cbbd03a222b9', '.gfid/1f003902-e95c-482f-8828-187ba8a1a88d', '.gfid/40dc07f6-70b1-404e-9620-85e260d3cc34', '.gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2', '.gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586', '.gfid/765f5767-eea9-4f23-8cdb-d58d69ee360f', '.gfid/06df7912-0af2-4539-9ef3-21da4cbb7715', '.gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d', '.gfid/46253285-a7ba-437f-a27c-ad281a39eda2'}}]
[2021-12-01 19:29:05.581610] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/94949690-4829-46d6-8feb-229525fa1a3a}]
[2021-12-01 19:29:05.581773] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/6ef035fe-99dc-42e1-94e8-cbbd03a222b9}]
[2021-12-01 19:29:05.581904] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/1f003902-e95c-482f-8828-187ba8a1a88d}]
[2021-12-01 19:29:05.582021] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/40dc07f6-70b1-404e-9620-85e260d3cc34}]
[2021-12-01 19:29:05.582132] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2}]
[2021-12-01 19:29:05.582240] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586}]
[2021-12-01 19:29:05.582348] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/765f5767-eea9-4f23-8cdb-d58d69ee360f}]
[2021-12-01 19:29:05.582454] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/06df7912-0af2-4539-9ef3-21da4cbb7715}]
[2021-12-01 19:29:05.582560] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d}]
[2021-12-01 19:29:05.582664] D [master(worker /nodirectwritedata/gluster/gvol0):319:a_syncdata] _GMaster: candidate for syncing [{file=.gfid/46253285-a7ba-437f-a27c-ad281a39eda2}]
[2021-12-01 19:29:05.630863] D [resource(worker /nodirectwritedata/gluster/gvol0):1442:rsync] SSH: files: .gfid/94949690-4829-46d6-8feb-229525fa1a3a, .gfid/6ef035fe-99dc-42e1-94e8-cbbd03a222b9, .gfid/1f003902-e95c-482f-8828-187ba8a1a88d, .gfid/40dc07f6-70b1-404e-9620-85e260d3cc34, .gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2, .gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586, .gfid/765f5767-eea9-4f23-8cdb-d58d69ee360f, .gfid/06df7912-0af2-4539-9ef3-21da4cbb7715, .gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d, .gfid/46253285-a7ba-437f-a27c-ad281a39eda2
[2021-12-01 19:29:05.802701] I [master(worker /nodirectwritedata/gluster/gvol0):1996:syncjob] Syncer: Sync Time Taken [{job=1}, {num_files=10}, {return_code=0}, {duration=0.1716}]
[2021-12-01 19:29:05.803521] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/94949690-4829-46d6-8feb-229525fa1a3a}]
[2021-12-01 19:29:05.803777] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/6ef035fe-99dc-42e1-94e8-cbbd03a222b9}]
[2021-12-01 19:29:05.803934] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/1f003902-e95c-482f-8828-187ba8a1a88d}]
[2021-12-01 19:29:05.804063] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/40dc07f6-70b1-404e-9620-85e260d3cc34}]
[2021-12-01 19:29:05.804182] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/5c18900a-9097-44c9-8bf6-15f84a9d99a2}]
[2021-12-01 19:29:05.804295] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/706380df-bed2-4a6b-8b3d-dcda90ffc586}]
[2021-12-01 19:29:05.804406] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/765f5767-eea9-4f23-8cdb-d58d69ee360f}]
[2021-12-01 19:29:05.804540] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/06df7912-0af2-4539-9ef3-21da4cbb7715}]
[2021-12-01 19:29:05.804665] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/a58b39e1-7a07-4250-bb1d-f8714fbaeb4d}]
[2021-12-01 19:29:05.804792] D [master(worker /nodirectwritedata/gluster/gvol0):325:regjob] _GMaster: synced [{file=.gfid/46253285-a7ba-437f-a27c-ad281a39eda2}]
[2021-12-01 19:29:05.817443] I [master(worker /nodirectwritedata/gluster/gvol0):1422:process] _GMaster: Entry Time Taken [{UNL=1}, {RMD=0}, {CRE=0}, {MKN=0}, {MKD=0}, {REN=0}, {LIN=0}, {SYM=0}, {duration=0.0545}]
[2021-12-01 19:29:05.817664] I [master(worker /nodirectwritedata/gluster/gvol0):1432:process] _GMaster: Data/Metadata Time Taken [{SETA=0}, {meta_duration=0.0000}, {SETX=0}, {XATT=0}, {DATA=10}, {data_duration=0.2363}]
[2021-12-01 19:29:05.818019] I [master(worker /nodirectwritedata/gluster/gvol0):1442:process] _GMaster: Batch Completed [{mode=live_changelog}, {duration=0.3024}, {changelog_start=1638386945}, {changelog_end=1638386945}, {num_changelogs=1}, {stime=(1638386944, 0)}, {entry_stime=(1638386944, 0)}]
[2021-12-01 19:29:10.823303] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16085 done, took 5.313680 seconds
[2021-12-01 19:29:15.836146] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16086 done, took 5.012481 seconds
[2021-12-01 19:29:20.849710] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16087 done, took 5.013163 seconds
[2021-12-01 19:29:25.860990] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16088 done, took 5.010890 seconds
[2021-12-01 19:29:30.871228] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16089 done, took 5.009902 seconds
[2021-12-01 19:29:35.882377] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16090 done, took 5.010731 seconds
[2021-12-01 19:29:40.895632] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16091 done, took 5.012887 seconds
[2021-12-01 19:29:45.908611] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16092 done, took 5.012609 seconds
[2021-12-01 19:29:45.908954] D [master(worker /nodirectwritedata/gluster/gvol0):561:crawlwrap] _GMaster: 12 crawls, 2 turns
[2021-12-01 19:29:50.922357] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16093 done, took 5.013328 seconds
[2021-12-01 19:29:55.932655] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16094 done, took 5.009902 seconds
[2021-12-01 19:30:00.945602] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16095 done, took 5.012597 seconds
[2021-12-01 19:30:05.589508] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140055754548992:1638387005.5893202 keep_alive({'version': (1, 0), 'uuid': '752ceb0d-77c2-42c5-9ba1-be43b34af460', 'retval': 0, 'volume_mark': (1621660285, 623270), 'timeout': 1638387125},) ...
[2021-12-01 19:30:05.609650] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140055754548992:1638387005.5893202 keep_alive -> 42375
[2021-12-01 19:30:06.18029] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16096 done, took 5.072013 seconds
[2021-12-01 19:30:11.31426] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16097 done, took 5.013012 seconds
[2021-12-01 19:30:16.41805] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16098 done, took 5.010033 seconds
[2021-12-01 19:30:21.54850] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16099 done, took 5.012648 seconds
[2021-12-01 19:30:26.68366] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16100 done, took 5.013113 seconds
[2021-12-01 19:30:31.81835] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16101 done, took 5.013060 seconds
[2021-12-01 19:30:36.96827] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16102 done, took 5.013379 seconds
[2021-12-01 19:30:41.112705] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16103 done, took 5.013963 seconds
[2021-12-01 19:30:46.125863] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16104 done, took 5.012781 seconds
[2021-12-01 19:30:46.126178] D [master(worker /nodirectwritedata/gluster/gvol0):561:crawlwrap] _GMaster: 12 crawls, 0 turns
[2021-12-01 19:30:46.903435] D [gsyncd(config-get):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:30:47.78415] D [gsyncd(status):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:30:48.729136] D [gsyncd(config-get):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:30:48.918635] D [gsyncd(status):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:30:51.139384] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16105 done, took 5.013120 seconds
[2021-12-01 19:30:52.186490] D [gsyncd(config-get):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:30:52.374271] D [gsyncd(status):303:main] <top>: Using session config file [{path=/var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf}]
[2021-12-01 19:30:56.153017] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16106 done, took 5.013209 seconds
[2021-12-01 19:31:01.163733] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16107 done, took 5.010355 seconds
[2021-12-01 19:31:05.670277] D [repce(worker /nodirectwritedata/gluster/gvol0):195:push] RepceClient: call 24892:140055754548992:1638387065.6700802 keep_alive({'version': (1, 0), 'uuid': '752ceb0d-77c2-42c5-9ba1-be43b34af460', 'retval': 0, 'volume_mark': (1621660285, 623270), 'timeout': 1638387185},) ...
[2021-12-01 19:31:05.690502] D [repce(worker /nodirectwritedata/gluster/gvol0):215:__call__] RepceClient: call 24892:140055754548992:1638387065.6700802 keep_alive -> 42376
[2021-12-01 19:31:06.176801] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16108 done, took 5.012715 seconds
[2021-12-01 19:31:11.190548] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16109 done, took 5.013036 seconds
[2021-12-01 19:31:16.205242] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16110 done, took 5.013861 seconds
[2021-12-01 19:31:21.218741] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16111 done, took 5.013079 seconds
[2021-12-01 19:31:26.232336] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16112 done, took 5.013160 seconds
[2021-12-01 19:31:31.245228] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16113 done, took 5.012511 seconds
[2021-12-01 19:31:36.258789] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16114 done, took 5.013165 seconds
[2021-12-01 19:31:41.272858] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16115 done, took 5.013644 seconds
[2021-12-01 19:31:46.291486] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16116 done, took 5.018257 seconds
[2021-12-01 19:31:46.291796] D [master(worker /nodirectwritedata/gluster/gvol0):561:crawlwrap] _GMaster: 12 crawls, 0 turns
[2021-12-01 19:31:51.302130] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16117 done, took 5.010253 seconds
[2021-12-01 19:31:56.313253] D [master(worker /nodirectwritedata/gluster/gvol0):555:crawlwrap] _GMaster: ... crawl #16118 done, took 5.010716 seconds
hunter86bg commented 2 years ago

Gluster version: Master:

hi  glusterfs-client                       8.5-ubuntu1~bionic1                             amd64        clustered file-system (client package)
hi  glusterfs-common                       8.5-ubuntu1~bionic1                             amd64        GlusterFS common libraries and translator modules
hi  glusterfs-server                       8.5-ubuntu1~bionic1                             amd64        clustered file-system (server package)

Slave:

hi  glusterfs-client                       8.5-ubuntu1~bionic1                             amd64        clustered file-system (client package)
hi  glusterfs-common                       8.5-ubuntu1~bionic1                             amd64        GlusterFS common libraries and translator modules
hi  glusterfs-server                       8.5-ubuntu1~bionic1                             amd64        clustered file-system (server package)

Pystack output:

# pystack -v 24892;echo;echo;echo;echo;echo;echo;echo; pystack -v 25678
Standard Output:
[New LWP 24907]
[New LWP 24909]
[New LWP 24977]
[New LWP 24978]
[New LWP 24979]
[New LWP 24980]
[New LWP 24981]
[New LWP 24982]
[New LWP 24983]
[New LWP 24984]
[New LWP 24985]
[New LWP 24986]
[New LWP 24987]
[New LWP 24988]
[New LWP 24989]
[New LWP 24990]
[New LWP 24991]
[New LWP 24992]
[New LWP 24993]
[New LWP 24994]
[New LWP 24995]
[New LWP 24996]
[New LWP 24997]
[New LWP 25027]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f618effce1f in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fff22145870) at ../sysdeps/unix/sysv/linux/select.c:41
$1 = (void *) 0x1

Standard Error:
41  ../sysdeps/unix/sysv/linux/select.c: No such file or directory.

Dumping Threads....

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 445, in keep_alive
    time.sleep(gap)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 176, in listen
    select((self.inf,), (), ())
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 488, in select
    return eintr_wrap(oselect.select, oselect.error, *args)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 480, in eintr_wrap
    return func(*args)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 773, in tailer
    time.sleep(0.5)

---------------

  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in <module>
    main()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 317, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker
    local.service_loop(remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1316, in service_loop
    g2.crawlwrap()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 607, in crawlwrap
    time.sleep(self.sleep_interval)
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>

Standard Output:
[New LWP 25679]
[New LWP 25715]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f2b54b407c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f2b44000e70) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
$1 = (void *) 0x1

Standard Error:
205 ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.

Dumping Threads....

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 272, in wmon
    slave_host, master, suuid, slavenodes)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 245, in monitor
    ret = nwait(cpid)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 117, in nwait
    p2, r = waitpid(p, o)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 492, in waitpid
    return eintr_wrap(owaitpid, OSError, *args)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 480, in eintr_wrap
    return func(*args)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 769, in tailer
    [po.stderr for po in errstore], [], [], 1)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 488, in select
    return eintr_wrap(oselect.select, oselect.error, *args)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 480, in eintr_wrap
    return func(*args)

---------------

  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in <module>
    main()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 317, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 60, in subcmd_monitor
    return monitor.monitor(local, remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 360, in monitor
    return Monitor().multiplex(*distribute(local, remote))
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 296, in multiplex
    t.join()
  File "/usr/lib/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
Shwetha-Acharya commented 2 years ago

@hunter86bg can you please share the brick logs and mnt logs of slave and master at the same time?

Shwetha-Acharya commented 2 years ago

Is the issue still present? or is it randomly getting resolved as well?

hunter86bg commented 2 years ago

Usually we force it to move to another node by using iptables to block outgoing traffic to slave nodes. Last time it self resolved and become stuck again /no actions from our side/.

I can say it self resolves rarely.

Edit: It stucks , then self resolved than stucks again. I will check if the issue was "force-resolved" via the iptables method or if it still affects the setup. Edit2: it was "fixed", so we can't debug it right now.

hunter86bg commented 2 years ago

I will provide the logs as soon as I can.

hunter86bg commented 2 years ago

slave1.tar.gz master2.tar.gz

hunter86bg commented 2 years ago

The replication was (at that time) from master2 to slave1.

hunter86bg commented 2 years ago

@Shwetha-Acharya , thanks for taking a look at it. It looks strange when the logs do not provide useful info.

Shwetha-Acharya commented 2 years ago

Time of issues (aprox):

# TZ='UTC' date --date='@1638386942' Wed Dec 1 19:29:02 UTC 2021 Nothing significant is seen on the logs. Can you please specify the workload?

# TZ='UTC' date --date='@1638306241' Tue Nov 30 21:04:01 UTC 2021

There were multiple occurances of "No such file or directory" errors around that time.

[2021-11-29 14:24:43.000376] E [MSGID: 117053] [changelog-helpers.c:441:changelog_rollover_changelog] 0-gvol0-changelog: error unlinking empty changelog [{path=/nodirectwritedata/gluster/gvol0/.glusterfs/changelogs/CHANGELOG}, {errno=2}, {error=No such file or directory}]

Either /nodirectwritedata/gluster/gvol0/.glusterfs/changelogs/CHANGELOG was not present or was not accessable for some reason.

hunter86bg commented 2 years ago

Thanks @Shwetha-Acharya ,

I'll monitor that file and see what happens.

hunter86bg commented 2 years ago

@Shwetha-Acharya ,

as you have noticed the first time there was no indication of a problem in the logs (or at least I missed it). Yet, the following puzzles me. Shouldn't all passive nodes have the same content of the CHANGELOG file ?

master1
GlusterFS Changelog | version: v1.2 | encoding : 2
Da9232e02-f536-4ee4-92d9-96559cdf5dcaDea8dd9ce-1e8b-419b-b247-44e9306d96c9Ef545e611-22cf-4dea-bcf6-bf3a66769b16233318810811727651a20-531d-4891-a78e-496804edfd00/voicemail_1638912529_1638912523611980_77028.wavDf545e611-22cf-4dea-bcf6-bf3a66769b16E1d128108-0878-4b1a-880b-69b7c46d38cc2333152108117c01f79a7-22e9-492d-9086-3e36a1fef50f/.lock-99490825E1d128108-0878-4b1a-880b-69b7c46d38cc9c01f79a7-22e9-492d-9086-3e36a1fef50f/.lockE1d128108-0878-4b1a-880b-69b7c46d38cc5c01f79a7-22e9-492d-9086-3e36a1fef50f/.lock-99490825E1d128108-0878-4b1a-880b-69b7c46d38cc5c01f79a7-22e9-492d-9086-3e36a1fef50f/.lockE0d2e3fad-d79b-4b26-b561-6c0f229a91cf2333152108117c01f79a7-22e9-492d-9086-3e36a1fef50f/.lock-03494068E0d2e3fad-d79b-4b26-b561-6c0f229a91cf9c01f79a7-22e9-492d-9086-3e36a1fef50f/.lockE0d2e3fad-d79b-4b26-b561-6c0f229a91cf5c01f79a7-22e9-492d-9086-3e36a1fef50f/.lock-03494068E0d2e3fad-d79b-4b26-b561-6c0f229a91cf5c01f79a7-22e9-492d-9086-3e36a1fef50f/.lockD9b980df6-ae7a-48c8-bd14-73825677ed5a

master
GlusterFS Changelog | version: v1.2 | encoding : 2

master3
GlusterFS Changelog | version: v1.2 | encoding : 2
D9b980df6-ae7a-48c8-bd14-73825677ed5aDf545e611-22cf-4dea-bcf6-bf3a66769b16Da9232e02-f536-4ee4-92d9-96559cdf5dcaEbfc973b9-4ecc-459d-b9dc-b25f43cfc2de23331881081177e64639b-c69f-43f9-afb0-507ce1ee1ebd/record_1638912536507713_52585-in.gsmE310ba9f1-2d5b-4456-9922-4ef834e95ee723331881081177e64639b-c69f-43f9-afb0-507ce1ee1ebd/record_1638912536507713_52585-out.gsm

P.S.: Currently active node is master3 (after the manual "failover")

aravindavk commented 2 years ago

Shouldn't all passive nodes have the same content of the CHANGELOG file ?

Order per file will be same across the nodes. But the operations can get recorded in different changelog files.

hunter86bg commented 2 years ago

It seems that the problem is related to the changelogs.

Here is example of the past 2 issues (timestamps from first comment):

[2021-11-30 21:04:30.114284] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 92962 [/nodirectwritedata/gluster/gvol0] (time: 1638306270.114149), (vec: 1, len: 4104) 
[2021-11-30 21:04:44.129055] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 92963 [/nodirectwritedata/gluster/gvol0] (time: 1638306284.128932), (vec: 1, len: 4104) 
[2021-11-30 21:04:59.144783] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 92964 [/nodirectwritedata/gluster/gvol0] (time: 1638306299.144665), (vec: 1, len: 4104) 
[2021-11-30 21:05:09.190179] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1638306276}, {end=1638306309}] 
[2021-11-30 21:05:09.190320] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1633926166}, {max=1638306299}, {total_changelogs=195980}] 
[2021-11-30 21:05:09.190562] I [MSGID: 132036] [gf-history-changelog.c:955:gf_history_changelog] 0-gfchangelog: FINAL [{from=1638306284}, {to=1638306299}, {changes=2}] 
[2021-11-30 21:05:09.190890] D [MSGID: 0] [gf-history-changelog.c:297:gf_history_changelog_scan] 0-gfchangelog: hist_done 1, is_last_scan: 0 
[2021-11-30 21:05:10.191050] D [MSGID: 0] [gf-history-changelog.c:297:gf_history_changelog_scan] 0-gfchangelog: hist_done 0, is_last_scan: 1 
[2021-11-30 21:05:14.160553] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 92965 [/nodirectwritedata/gluster/gvol0] (time: 1638306314.160395), (vec: 1, len: 4104) 
[2021-11-30 21:05:28.175328] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 92966 [/nodirectwritedata/gluster/gvol0] (time: 1638306328.175190), (vec: 1, len: 4104) 
[2021-11-30 21:05:42.189783] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 92967 [/nodirectwritedata/gluster/gvol0] (time: 1638306342.189649), (vec: 1, len: 4104) 
[2021-11-30 21:05:56.204524] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 92968 [/nodirectwritedata/gluster/gvol0] (time: 1638306356.204377), (vec: 1, len: 4104)

[2021-12-01 19:14:57.071507] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95545 [/nodirectwritedata/gluster/gvol0] (time: 1638386097.71352), (vec: 1, len: 4104) 
[2021-12-01 19:15:11.086155] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95546 [/nodirectwritedata/gluster/gvol0] (time: 1638386111.86018), (vec: 1, len: 4104) 
[2021-12-01 19:15:25.100922] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95547 [/nodirectwritedata/gluster/gvol0] (time: 1638386125.100767), (vec: 1, len: 4104) 
[2021-12-01 19:28:37.165759] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95548 [/nodirectwritedata/gluster/gvol0] (time: 1638386917.165581), (vec: 1, len: 4104) 
[2021-12-01 19:28:51.180591] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95549 [/nodirectwritedata/gluster/gvol0] (time: 1638386931.180414), (vec: 1, len: 4104) 
[2021-12-01 19:28:55.481728] D [MSGID: 0] [gf-changelog-api.c:55:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/gluster/gsyncd/gvol0_nvfs30_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1638386931 to processed directory 
[2021-12-01 19:29:05.194963] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95550 [/nodirectwritedata/gluster/gvol0] (time: 1638386945.194678), (vec: 1, len: 4104) 
[2021-12-01 19:29:05.806899] D [MSGID: 0] [gf-changelog-api.c:55:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/gluster/gsyncd/gvol0_nvfs30_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1638386945 to processed directory 
[2021-12-02 00:00:00.037913] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95551 [/nodirectwritedata/gluster/gvol0] (time: 1638403200.37668), (vec: 1, len: 4104) 
[2021-12-02 00:00:14.051495] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95552 [/nodirectwritedata/gluster/gvol0] (time: 1638403214.51403), (vec: 1, len: 4104) 
[2021-12-02 00:00:29.065960] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 95553 [/nodirectwritedata/gluster/gvol0] (time: 1638403229.65853), (vec: 1, len: 4104)

Yet , again the geo-rep switched to master2 and again the problem:

#Timestamp:
# TZ='UTC' date --date='@1639061941'
Thu Dec  9 14:59:01 UTC 2021
### Note: Timestamps are in local time (PST):
# ls -ltr changes-nodirectwritedata-gluster-gvol0.log*
-rw------- 1 root root  184304 Sep  4 06:24 changes-nodirectwritedata-gluster-gvol0.log.7.gz
-rw------- 1 root root  302160 Sep  5 00:52 changes-nodirectwritedata-gluster-gvol0.log.6.gz
-rw------- 1 root root  344924 Sep 16 06:25 changes-nodirectwritedata-gluster-gvol0.log.5.gz
-rw------- 1 root root  785825 Sep 18 06:25 changes-nodirectwritedata-gluster-gvol0.log.4.gz
-rw------- 1 root root  222834 Oct 10 20:22 changes-nodirectwritedata-gluster-gvol0.log.3.gz
-rw------- 1 root root 1264993 Dec  5 22:31 changes-nodirectwritedata-gluster-gvol0.log.2.gz
-rw------- 1 root root       0 Dec  6 06:25 changes-nodirectwritedata-gluster-gvol0.log
-rw------- 1 root root  962913 Dec  9 11:54 changes-nodirectwritedata-gluster-gvol0.log.1
# tail -n 20 changes-nodirectwritedata-gluster-gvol0.log.1
[2021-12-09 18:17:56.053980] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105672 [/nodirectwritedata/gluster/gvol0] (time: 1639073876.53847), (vec: 1, len: 4104) 
[2021-12-09 18:18:10.068719] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105673 [/nodirectwritedata/gluster/gvol0] (time: 1639073890.68569), (vec: 1, len: 4104) 
[2021-12-09 18:18:24.083527] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105674 [/nodirectwritedata/gluster/gvol0] (time: 1639073904.83329), (vec: 1, len: 4104) 
[2021-12-09 18:24:18.208570] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105675 [/nodirectwritedata/gluster/gvol0] (time: 1639074258.208382), (vec: 1, len: 4104) 
[2021-12-09 18:24:32.223213] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105676 [/nodirectwritedata/gluster/gvol0] (time: 1639074272.223109), (vec: 1, len: 4104) 
[2021-12-09 18:24:46.238050] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105677 [/nodirectwritedata/gluster/gvol0] (time: 1639074286.237936), (vec: 1, len: 4104) 
[2021-12-09 18:25:00.252815] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105678 [/nodirectwritedata/gluster/gvol0] (time: 1639074300.252692), (vec: 1, len: 4104) 
[2021-12-09 18:25:14.267504] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105679 [/nodirectwritedata/gluster/gvol0] (time: 1639074314.267397), (vec: 1, len: 4104) 
[2021-12-09 18:25:18.774405] D [MSGID: 0] [gf-changelog-api.c:55:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/gluster/gsyncd/gvol0_nvfs30_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1639074314 to processed directory 
[2021-12-09 18:26:24.341382] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105680 [/nodirectwritedata/gluster/gvol0] (time: 1639074384.341183), (vec: 1, len: 4104) 
[2021-12-09 18:26:38.356096] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105681 [/nodirectwritedata/gluster/gvol0] (time: 1639074398.355991), (vec: 1, len: 4104) 
[2021-12-09 18:26:52.370825] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105682 [/nodirectwritedata/gluster/gvol0] (time: 1639074412.370690), (vec: 1, len: 4104) 
[2021-12-09 18:27:06.385413] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105683 [/nodirectwritedata/gluster/gvol0] (time: 1639074426.385269), (vec: 1, len: 4104) 
[2021-12-09 18:27:09.476805] D [MSGID: 0] [gf-changelog-api.c:55:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/gluster/gsyncd/gvol0_nvfs30_gvol0/nodirectwritedata-gluster-gvol0/.processing/CHANGELOG.1639074426 to processed directory 
[2021-12-09 19:51:37.640918] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105684 [/nodirectwritedata/gluster/gvol0] (time: 1639079497.640742), (vec: 1, len: 4104) 
[2021-12-09 19:51:51.655674] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105685 [/nodirectwritedata/gluster/gvol0] (time: 1639079511.655483), (vec: 1, len: 4104) 
[2021-12-09 19:52:05.669678] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105686 [/nodirectwritedata/gluster/gvol0] (time: 1639079525.669546), (vec: 1, len: 4104) 
[2021-12-09 19:54:25.055451] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105687 [/nodirectwritedata/gluster/gvol0] (time: 1639079665.55216), (vec: 1, len: 4104) 
[2021-12-09 19:54:39.069114] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105688 [/nodirectwritedata/gluster/gvol0] (time: 1639079679.68953), (vec: 1, len: 4104) 
[2021-12-09 19:54:53.083845] D [MSGID: 0] [gf-changelog-reborp.c:355:gf_changelog_event_handler] 0-gfchangelog: seq: 105689 [/nodirectwritedata/gluster/gvol0] (time: 1639079693.83746), (vec: 1, len: 4104)
root@master2:~# ps aux | grep [g]syncd.py
root     25678  0.0  0.0 229416 16332 ?        Ssl  Nov01   2:56 /usr/bin/python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --path=/nodirectwritedata/gluster/gvol0  --monitor -c /var/lib/glusterd/geo-replication/gvol0_slave3_gvol0/gsyncd.conf --iprefix=/var :gvol0 --glusterd-uuid=74927c6f-6c07-4814-86ea-c8b2070cf66b slave3::gvol0
root     48348  0.1  0.0 1823456 22500 ?       Sl   Dec05   6:01 python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py worker gvol0 slave3::gvol0 --feedback-fd 10 --local-path /nodirectwritedata/gluster/gvol0 --local-node master2 --local-node-id 74927c6f-6c07-4814-86ea-c8b2070cf66b --slave-id 190a1403-7135-48cb-8d8e-1b1573a61904 --subvol-num 1 --resource-remote slave2 --resource-remote-id 06a32508-7a10-4ff4-bb36-3c1ba3c248d1
root@master2:~# source ~/venv/bin/activate
(venv) root@master2:~# pystack -v 25678; echo;echo;echo;pystack -v 48348

(venv) root@master2:~# pystack -v 25678; echo;echo;echo;pystack -v 48348
Standard Output:
[New LWP 25679]
[New LWP 25715]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f2b54b407c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f2b44000e70) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
$1 = (void *) 0x1

Standard Error:
205 ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.

Dumping Threads....

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 272, in wmon
    slave_host, master, suuid, slavenodes)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 245, in monitor
    ret = nwait(cpid)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 117, in nwait
    p2, r = waitpid(p, o)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 492, in waitpid
    return eintr_wrap(owaitpid, OSError, *args)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 480, in eintr_wrap
    return func(*args)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 769, in tailer
    [po.stderr for po in errstore], [], [], 1)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 488, in select
    return eintr_wrap(oselect.select, oselect.error, *args)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 480, in eintr_wrap
    return func(*args)

---------------

  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in <module>
    main()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 317, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 60, in subcmd_monitor
    return monitor.monitor(local, remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 360, in monitor
    return Monitor().multiplex(*distribute(local, remote))
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 296, in multiplex
    t.join()
  File "/usr/lib/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>

Standard Output:
[New LWP 48363]
[New LWP 48365]
[New LWP 48424]
[New LWP 48425]
[New LWP 48426]
[New LWP 48427]
[New LWP 48428]
[New LWP 48429]
[New LWP 48430]
[New LWP 48431]
[New LWP 48432]
[New LWP 48433]
[New LWP 48434]
[New LWP 48435]
[New LWP 48436]
[New LWP 48437]
[New LWP 48439]
[New LWP 48440]
[New LWP 48441]
[New LWP 48442]
[New LWP 48443]
[New LWP 48444]
[New LWP 48445]
[New LWP 48470]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fc929c37e1f in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fff6b7f8360) at ../sysdeps/unix/sysv/linux/select.c:41
$1 = (void *) 0x1

Standard Error:
41  ../sysdeps/unix/sysv/linux/select.c: No such file or directory.

Dumping Threads....

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 445, in keep_alive
    time.sleep(gap)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1988, in syncjob
    time.sleep(0.5)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 176, in listen
    select((self.inf,), (), ())
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 488, in select
    return eintr_wrap(oselect.select, oselect.error, *args)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 480, in eintr_wrap
    return func(*args)

---------------

  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 393, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 773, in tailer
    time.sleep(0.5)

---------------

  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in <module>
    main()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 317, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker
    local.service_loop(remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1316, in service_loop
    g2.crawlwrap()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 607, in crawlwrap
    time.sleep(self.sleep_interval)
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
hunter86bg commented 2 years ago

@aravindavk , @Shwetha-Acharya ,

any idea how to debug the changelog issue ?

Shwetha-Acharya commented 2 years ago

@hunter86bg you can check rsync logs to find if there is anything going wrong with respect to rsync with the command: gluster volume geo-replication :: config rsync-options '-vv --logfile=/var/log/glusterfs/geo-replication/respective session directory/rsync log file name'

Also please specify the rsync version you are using.

hunter86bg commented 2 years ago

Hi Shwetha, thanks for the hint. I will enable the logs as soon as I'm allowed to.

Rsync is the same version on both master and slave nodes:

rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.