search

NCAR / DART_CASES

DART CASE directories from CESM experiments.
0 stars 3 forks source link

Permissions of files migrated to Campaign Storage #60

Open kdraeder opened 4 years ago

kdraeder commented 4 years ago

This has become an issue because files are being migrated to the RDA from Campaign Storage by people outside our group. They need global read permission on the files and directories in /gpfs/csfs1/cisl/dares/Reanalyses/f.e21.FHIST_BGC.f09_025.CAM6assim.011 I've been migrating files to CS using globus because that was the only way at the beginning, it has automatic failure recovery, and it saves some core hours by moving the files asynchronously after the submission job ends. But globus generally can't control the permissions of the files it creates. They are created according to the umask of the destination environment. I don't know how to access the umask used for CS. See https://docs.globus.org/faq/transfer-sharing/

how_do_i_control_file_permissions_during_transfers

The suggestions on that site didn't look appealing. So our files on campaign storage end up without global read permission.

The simplest and probably least reliable solution is to manually change the permissions after each complete year is copied to CS. Unless we want the migration of data to RDA to happen more frequently, which would require more frequent manual changes.

It would be great to include a few lines in the Reanalyis scripts to change the permissions automatically, but at least some of the changes need to be made after all of the current scripts have finished.

So some of the options seem to be: 1) Abandon globus and recode using rsync(?) and any needed error recovery, and pay the extra core-hours. 2) Use globus and manually update the permissions as needed. 3) Add an additional script to the workflow that would change the permissions, which would be run manually as needed.

Other ideas? Context?

timhoar commented 4 years ago

After reading the globus docs you mention, the documentation mentions

By setting ACLs on the destination such that they are applied to all new files in a directory tree, you can effectively set ACLs on the files created by the GridFTP server

so it sounds like we need to explore access control lists ... which (rather unbelievably) came out today in the CISL Daily Bulletin 'Tip': https://www2.cisl.ucar.edu/resources/storage-and-file-systems/glade/using-access-control-lists I have not read this, and I don't know what what machine (data-access?) you would use to install the ACL, but it's another idea.

kdraeder commented 4 years ago

Thanks for the tip! I'll look through the Daily B article. I also don't know what machine to use. I could ask Chi-fan about that.

kdraeder commented 4 years ago

Following Tim's suggestion; DailyB ACL article:

We want the section dealing with default ACLs: To enable another user to read and navigate into a directory

setfacl --modify user:bjsmith:r-x my_shared_directory

To set the directory's default ACL so that any new files or subdirectories automatically have those same permissions,

setfacl --modify default:user:bjsmith:r-x my_shared_directory ls -ld my_shared_directory drwxr-x---+ 2 shiquan ncar 4096 Mar 6 10:12 my_shared_directory The '+' says that there's an ACL for this directory.

We may also want syntax to give access to a group; replace user:bjsmith:r-x with group:csg:r-x

getfacl shows the ACL, which shows bjsmith as another 'user', not part of the group, despite the appearance of the permissions.

I'll try to implement this after I find out from Chi-Fan who needs permission.