Should BB API preserve permissions & xattrs to destination file

[tonyhutter]

Describe the bug Should the BB API preserve permissions and xattrs on the files it's transferring?

I used your test_basic_xfer utility to copy a file on the burst buffer with O777 permissions, and xattr "hello=world" set, to its destination on GPFS (with umask set to u=rwx,g=rwx,o=rwx). After the transfer the destination file was O750, not O777, and the xattr was not set.

To Reproduce Make this change:

--- test_basic_xfer.c.orig  2020-05-20 15:26:40.926135000 -0700
+++ test_basic_xfer.c   2020-10-08 12:27:10.670129000 -0700
@@ -230,7 +230,7 @@

     if(dogenerate)
     {
-        snprintf(cmd, sizeof(cmd), "/opt/ibm/bb/tools/randfile --file=%s --by 512 --size=%ld", sfn, filesize);
+        snprintf(cmd, sizeof(cmd), "/opt/ibm/bb/tools/randfile --file=%s --by 512 --size=%ld && chmod 777 %s && attr -s hello -V world %s", sfn, filesize, sfn, sfn);
         if(rank == 0)
         {
             printf("Generate random file: %s\n", cmd);

Then:

# Check that destination file doesn't already exist
$ ls -l /p/gpfs1/hutter2/dst/rank.0
ls: cannot access /p/gpfs1/hutter2/dst/rank.0: No such file or directory

# Check umask
$ umask -S
u=rwx,g=rwx,o=rwx

# Do the transfer
$ ./test_basic_xfer 1 /p/gpfs1/hutter2/dst 1024
Source file: /tmp/bblv_hutter2_130450//rank.0
Target file: /p/gpfs1/hutter2/dst/rank.0
METADATA branch=(null)
METADATA gitcommit=(null)
METADATA job=(null)
METADATA type=out
METADATA jobid=130450
METADATA jobstep=2768240641
METADATA user=hutter2
METADATA pfspath=/p/gpfs1/hutter2/dst
METADATA bbpath=/tmp/bblv_hutter2_130450/
METADATA execname=./test_basic_xfer
METADATA filesize=1024
METADATA ranks=1
Generate random file: /opt/ibm/bb/tools/randfile --file=/tmp/bblv_hutter2_130450//rank.0 --by 512 --size=1024 && chmod 777 /tmp/bblv_hutter2_130450//rank.0 && attr -s hello -V world /tmp/bblv_hutter2_130450//rank.0
Attribute "hello" set to a 5 byte value for /tmp/bblv_hutter2_130450//rank.0:
world
Creating transfer definition
Adding files to transfer definition
Obtaining transfer handle
PERF(1,/p/gpfs1/hutter2/dst,1024 x 1):  BB_GetTransferHandle took 0.012419 seconds
METRIC bbGetTransferHandle_time=0.012419
Starting transfer
PERF(1,/p/gpfs1/hutter2/dst,1024 x 1):  BB_StartTransfer took 0.113452 seconds
METRIC bbStartTransfer_time=0.113452
PERF(1,/p/gpfs1/hutter2/dst,1024 x 1):  Transfer took 1.05463 seconds (0.000925976 MiBps)
METRIC bbTransfer_time=1.05463
METRIC bbTransfer_bandwidth=970.956
Terminating BB library

# Verify source and destination permissions
$ ls -l /tmp/bblv_hutter2_130450//rank.0
-rwxrwxrwx 1 hutter2 hutter2 1024 Oct  8 12:27 /tmp/bblv_hutter2_130450//rank.0

$ ls -l /p/gpfs1/hutter2/dst/rank.0
-rwxr-x--- 1 hutter2 hutter2 1024 Oct  8 12:27 /p/gpfs1/hutter2/dst/rank.0

# Verify source and destination xattr
$ attr -g hello /tmp/bblv_hutter2_130450//rank.0
Attribute "hello" had a 5 byte value for /tmp/bblv_hutter2_130450//rank.0:
world

$ attr -g hello /p/gpfs1/hutter2/dst/rank.0
attr_get: No data available
Could not get "hello" for /p/gpfs1/hutter2/dst/rank.0

Expected behavior ?

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Machine [LLNL]
• Version [ibm-burstbuffer 1.7.2-3094]

Additional context

Issue Source:

[tonyhutter]

Another observation: if you create the destination file in advance of transferring it, it will preserve the permissions of the destination file. That is, the transfer doesn't change the permission bits on the destination file if the destination file already exists.

[tgooding]

Yes, it should be non-extended ACL preserving in both transfer directions. We'll try to recreate with the current software. (as far as extended attributes, my understanding those were disabled in your PFS config)

[meahoibm]

Was ./test_basic_xfer 1 /p/gpfs1/hutter2/dst 1024 run on a compute node? What is the umask on the node with bbserver?

[tonyhutter]

Yes, it was run on the compute node. umask was:

# Check umask
$ umask -S
u=rwx,g=rwx,o=rwx

[tonyhutter]

@meahoibm sorry, I gave you the umask on the compute node, not the bbserver node. Let me see if I can get that info...

[tonyhutter]

The bbserver umask is: u=rwx,g=rx,o=rx

I would think the bbserver's umask shouldn't make a difference though, as I'm copying existing files, not creating new ones.

[meahoibm]

Just wanted the umask as an environment consideration in looking at the burst buffer code handling of the copy execution. I would expect rwx/ugo preservation on copy.

[meahoibm]

The umask we have on our system for bash/terminal session is 022. When I cp a file in my home directory with rwxrwxrwx to /tmp or to GPFS the file changes to rwxr-xr-x if not pre-existing. Do you encounter the same? The process umask on bbServer process is 027 by the default service definition. This affects the ugo bits on any freshly created target file.
The umask can be changed in the bbserver service definition by an administrator. For reference, I had checked like this: grep -i umask /usr/lib/systemd/system/bbserver.service UMask=0027

[tonyhutter]

Ok, I was unaware of the umask setting in /usr/lib/systemd/system/bbserver.service. That would match the results I'm seeing. Thanks for the help. Closing issue