ccy
commented
1 year ago I found a workaround solution here. I suspect this is a bug happen to intermediate zone when performing rsync task in cascading geo-replication setup.
When I update last byte of the file in Zone A, and geo-replication task start:
# Show .processing directory
$ ls /var/lib/misc/gluster/gsyncd/..././.processing
total 4
-rw-r--r-- 1 root root 39 Jul 27 21:42 CHANGELOG.1690465326
# Show content of CHANGLOG file
$ cat CHANGELOG.1690465326
D c8b7999b-78ef-4894-b35a-4b413070194cThis is the log file content showing rsync shall perform update of 1 file (in Number of regular files transferred):
[2023-07-27 13:42:10.323244] D [primary(worker /mnt/brick/b):317:a_syncdata] _GPrimary: files [{files={'.gfid/c8b7999b-78ef-4894-b35a-4b413070194c'}}]
[2023-07-27 13:42:10.323301] D [primary(worker /mnt/brick/b):320:a_syncdata] _GPrimary: candidate for syncing [{file=.gfid/c8b7999b-78ef-4894-b35a-4b413070194c}]
[2023-07-27 13:42:10.353568] D [resource(worker /mnt/brick/b):1442:rsync] SSH: files: .gfid/c8b7999b-78ef-4894-b35a-4b413070194c
[2023-07-27 13:42:39.128166] I [resource(worker /mnt/brick/b):1530:rsync] SSH: rsync verbose [{
data='building file list ... done
.gfid/c8b7999b-78ef-4894-b35a-4b413070194c
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 5,842,186,240 bytes
Total transferred file size: 5,842,186,240 bytes
Literal data: 29,888 bytes
Matched data: 5,842,156,352 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 335,908
Total bytes received: 611,565
sent 335,908 bytes received 611,565 bytes 32,117.73 bytes/sec
total size is 5,842,186,240 speedup is 6,166.07
'}]
[2023-07-27 13:42:39.128545] D [primary(worker /mnt/brick/b):326:regjob] _GPrimary: synced [{file=.gfid/c8b7999b-78ef-4894-b35a-4b413070194c}]Next, the intermediate node of Zone B:
# Show .processing directory
$ ls /var/lib/misc/gluster/gsyncd/..././.processing
total 4
-rw-r--r-- 1 root root 48 Jul 27 21:50 CHANGELOG.1690465836
# Show content of CHANGLOG file
$ cat CHANGELOG.1690465836
D c8b7999b-78ef-4894-b35a-4b413070194c
M c8b7999b-78ef-4894-b35a-4b413070194c SETATTR
M 00000000-0000-0000-0000-000000000001 SETXATTRThis is the log file content showing rsync does not perform any update (in Number of regular files transferred). But the timestamp of the file do updated.
[{batch=['/var/lib/misc/gluster/gsyncd/b_gfs-c1_b/mnt-brick-b/.processing/CHANGELOG.1690465836']}]
[2023-07-27 13:50:38.544077] D [primary(worker /mnt/brick/b):1343:process] _GPrimary: processing change [{changelog=/var/lib/misc/gluster/gsyncd/b_gfs-c1_b/mnt-brick-b/.processing/CHANGELOG.1690465836}]
[2023-07-27 13:50:38.544209] D [primary(worker /mnt/brick/b):1207:process_change] _GPrimary: entries: []
[2023-07-27 13:50:38.545924] D [repce(worker /mnt/brick/b):195:push] RepceClient: call 2018616:139904444790592:1690465838.5458996 meta_ops([{'op': 'META', 'skip_entry': False, 'go': '.gfid/c8b7999b-78ef-4894-b35a-4b413070194c', 'stat': {'uid': 0, 'gid': 0, 'mode': 33188, 'atime': 1690355048.1178703, 'mtime': 1690465789.6717978}}],) ...
[2023-07-27 13:50:38.547595] D [repce(worker /mnt/brick/b):215:__call__] RepceClient: call 2018616:139904444790592:1690465838.5458996 meta_ops -> []
[2023-07-27 13:50:38.548894] D [primary(worker /mnt/brick/b):317:a_syncdata] _GPrimary: files [{files={'.gfid/c8b7999b-78ef-4894-b35a-4b413070194c'}}]
[2023-07-27 13:50:38.548938] D [primary(worker /mnt/brick/b):320:a_syncdata] _GPrimary: candidate for syncing [{file=.gfid/c8b7999b-78ef-4894-b35a-4b413070194c}]
[2023-07-27 13:50:38.842242] D [resource(worker /mnt/brick/b):1442:rsync] SSH: files: .gfid/c8b7999b-78ef-4894-b35a-4b413070194c
[2023-07-27 13:50:38.910505] I [resource(worker /mnt/brick/b):1530:rsync] SSH: rsync verbose [{
data='building file list ... done
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 5,842,186,240 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 216
Total bytes received: 17
sent 216 bytes received 17 bytes 466.00 bytes/sec
total size is 5,842,186,240 speedup is 25,073,760.69
'}]
[2023-07-27 13:50:38.910670] I [primary(worker /mnt/brick/b):2009:syncjob] Syncer: Sync Time Taken [{job=2}, {num_files=1}, {return_code=0}, {duration=0.0684}]Here are the summary:
# Zone A CHANGELOG
D c8b7999b-78ef-4894-b35a-4b413070194c
# Zone B CHANGELOG
D c8b7999b-78ef-4894-b35a-4b413070194c
M c8b7999b-78ef-4894-b35a-4b413070194c SETATTR
M 00000000-0000-0000-0000-000000000001 SETXATTRZone B file timestamp updated. Content also updated. Zone C file timestamp updated. Content does not update.
This is related to a weakness rsync delta transfer algorithm. Zone C's timestamp has updated due to this change log:
M c8b7999b-78ef-4894-b35a-4b413070194c SETATTR`before rsync operation:
rsync -v -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --xattrs --acls --existing ./ gfs-c1:/mnt/brick/bMy workaround solution is add rsync ignore-times flag (-I) to geo-replication config's rsync_options:
sudo gluster volume geo-replication $VOL $GEO_USER@$H2::$VOL config rsync_options '-I'
Description of problem:
I have three gluster zones and I create a gluster volume (
xfs) for each zone using the following layout:I then setup cascading geo-replication from Zone A -> Zone B -> Zone C.
I mount volume
xfsin Zone A, and dump a 5.5GB Windows 22H2 ISO file (sha1sum:a3f1a71bb41517e2ef07f09093bbcdca4f5c313c) to the mounting point. After a while I notice replication task dump the file from Zone A to Zone B, then Zone B to Zone C.Once the geo-replication task is completed, I use
sha1sumto check the file in three zone's volume and I get same sha1 hash value. So far so good.Next, I try to modify last byte (using
hexedit) in the file. I can notice the geo-replication from Zone A to Zone B always success, but Zone B to Zone C mostly fail to work. I wait overnight and hope it can propagate to C but it doesn't. I usesha1sumto check the file content for all 3 zones.If I update 1st or 2nd byte of the file, the geo-replication works by replication from zone A, B and C.
This log message keep show up in Zone B every minute:
And the directory
/mnt/brick/abc/.glusterfs/changelogs/2023/07/13will have new CHANGELOG fileThe exact command to reproduce the issue:
Use
hexeditto modify file.The full output of the command that failed:
Expected results: Cascading geo-replication should work even small change in source volume
Mandatory info: - The output of the
gluster volume infocommand:- The output of the
gluster volume statuscommand:- The output of the
gluster volume healcommand:**- Provide logs present on following locations of client and server nodes -
**- Is there any crash ? Provide the backtrace and coredump No crash happen.
Additional info:
- The operating system / glusterfs version: