Hidden directory is not being excluded

[gamename]

Hi Ganael,

I'm using this invocation: /usr/bin/fpsync -O '-x .glusterfs -x .root -x .r00t' -n 64 -f 50000 -vvv /ebs/geisprd/ /efs/nv-wc-geis-prd/vault11/

Both root and r00t are files. They are successfully excluded. But the hidden directory .glusterfs is not excluded. It is copied to the destination directory.

Is there something I'm doing wrong?

-Tennis

[martymac]

Hello Tennis,

No, it should be excluded too. I've successfully re-tested that kind of exclusion on my system and it works here :

$ mkdir -p /usr/src/.glusterfs ; touch /usr/src/.glusterfs/foobar ; mkdir -p /usr/src/foobar/.glusterfs ; touch /usr/src/foobar/.glusterfs/foobar
$ fpsync -O '-x .glusterfs -x .root -x .r00t' -n 64 -f 50000 -vvv /usr/src/ /var/tmp/src/
$ grep -ir -e foobar -e '\.glusterfs' /tmp/fpsync/parts/1596882201-69354
<no output>

Can you check that generated partitions really contain the .glusterfs directory ?

If in doubt, you can build fpart with debug enabled (--enable-debug configure option). You will get lots of output messages (when run fpart by hand) telling if files/directories are included or excluded :

valid_file(): checking name validity for directory: .glusterfs (path: /usr/src/.glusterfs)
valid_file(): directory: .glusterfs, validity: invalid
valid_file(): checking name validity for directory: .glusterfs (path: /usr/src/foobar/.glusterfs)
valid_file(): directory: .glusterfs, validity: invalid

Best regards,

Ganael.

[gamename]

Thanks, Ganael,

I'll give that a try and get back to you.

-Tennis

[gamename]

Hi Ganael,

Like you, we haven't been able to reproduce it on a small scale. But for large (multi TB) disks we do have some additional info.

For large file transfers, the exclude appears to mostly work. What we are seeing on the destination end is a copy of the hidden directory. BUT the contents of the hidden directory is tiny compared to the source hidden directory. In other words, only a portion of the hidden directory's contents is being transferred.

We are wondering if there are hard links involved to cause that. But that's just a theory. We haven't been able to prove it.

Thoughts? -Tennis

@themedina FYI, in case you want to chime in with additional info

[martymac]

Hello Tennis,

I haven't played with Glusterfs. Isn't that directory (.glusterfs) handled and re-created by Glusterfs itself ?

Does this hidden folder appear in the partition files generated by fpart ?

[gamename]

Hi Ganael,

Yes, .glusterfs is a directory created by Glusterfs. Glusterfs is not being used any longer, so it has no effect on the volume. The volume is created from a snapshot of the Gluster volume, but Gluster is not running on the host where it is mounted.

I'll find out if the hidden folder appears in the partition files generated by fpart.

[martymac]

Hi Tennis,

Any news ?

[gamename]

Sorry guy, no updates. The project we were working on that uses FPSync is probably on hold for now. If I can confirm this for you, I'll let you know. But there's no need to keep this open for now and have you continue to wait for us.