Add (even more) support for access control lists

[tschoonj]

Hi Bruce,

This fixes some file ACL trouble people encountered this morning at I20. Appears to be very similar to #27.

Best regards,

Tom

[newville]

@tschoonj @bruceravel I don't understand what problem this solves. We use Athena and other files over network-mounted shares (mixed linux/Windows) all the time and don't have problems that need to be solved with client code that aren't OS features like read/write permissions on files.

Why should Athena be expected to deal with such issues? Either it can open the file or it cannot.

[tschoonj]

@newville you may want to read the filetestpragma Perl docs to understand why we need this fix.

Basically, unlike Python and other languages, Perl's I/O functions do not support ACLs by default and need this pragma to activate support.

It is Diamond IT department's policy to use ACLs heavily throughout our network file systems, and I am sure they have good reasons to do so. I certainly cannot ask them to change this policy. All that matters to me (and my superiors) is that the I20 (and B18 last year) staff and users can open their data with Demeter at the beamline. This fix ensures that, keeping everybody here happy. I could have kept this fix in-house, but since I felt that others might benefit from it too, I opened this PR for Bruce to consider this fix for inclusion in the Demeter codebase.

[bruceravel]

Thanks, Tom.

[tschoonj]

You're most welcome. Thanks for merging it in.

[newville]

@tschoonj @bruceravel Actually, I still do not understand what problem this solves.

The docs for the filetest pragma imply that all file tests are fundamentally broken for networked filesystems in Perl, and that all use of '-r' etc tests must use this fix. That is, everywhere in the code, not just in this one file.

But, is that right? I honestly don't know. @tschoonj, ALL files have ACLs. Your remark of " It is department policy to use ACLs heavily throughout the network file system" can be shortened to "We use a filesystem". All files have ACLs. It should not be the job of the client program to have to know that multiple filesystems are in use. The client should be able to ask if it has permission to read/write a file, and be able to open the file for read and/or write: stat() and fopen(). I find it hard to believe that Perl has broken implementations of these.

If file tests are indeed broken for using networked filesystems, why does it work for me? We use networked file systems all the time. I can definitely open Athena project files across NFS and CIFS. We do this all the time, we use file permissions, and we have never seen a problem.

If this fix is needed, I do not understand why it only applies to one file.

[tschoonj]

The docs for the filetest pragma imply that all file tests are fundamentally broken for networked filesystems in Perl, and that all use of '-r' etc tests must use this fix. That is, everywhere in the code, not just in this one file.

That is indeed correct: ideally all files including perl filetests should receive this pragma. I agree that there is a strong case to be made that Perl's filetests are fundamentally broken, since these files can be opened with other applications.

But, is that right? I honestly don't know. @tschoonj, ALL files have ACLs. Your remark of " It is department policy to use ACLs heavily throughout the network file system" can be shortened to "We use a filesystem". All files have ACLs. It should not be the job of the client program to have to know that multiple filesystems are in use. The client should be able to ask if it has permission to read/write a file, and be able to open the file for read and/or write: stat() and fopen(). I find it hard to believe that Perl has broken implementations of these.

Not necessarily. There are two ways of managing permissions: the classic and most commonly used way happens via the (effective) user id and group id that a file has. This is what is used by the standard Perl tests, which rely internally on the stat function from the standard C library. Typically these permissions are managed via chown and chgrp. Some filesystems however, offer a second (and complementary to the first) way, called access control lists: these allow for a finer tuning of permissions that allow for example some groups to read and other groups to read and write. When the filetest pragma is in use, access() is used instead of stat(), which is capable of taking into account ACLs. ACLs are typically managed from the command line using setfacl and getfacl. ACLs are an entirely optional feature and can be turned on/off when mounting the filesystem.

If file tests are indeed broken for using networked filesystems, why does it work for me? We use networked file systems all the time. I can definitely open Athena project files across NFS and CIFS. We do this all the time, we use file permissions, and we have never seen a problem.

I guess ACLs are either not in use, or the combination of user id and group id is already favorable.

If this fix is needed, I do not understand why it only applies to one file.

Again you're right: ideally all files in Demeter should be checked for filetests, and if found, the pragma should be added...

[newville]

@tschoonj Yes, I understand what ACLs are. But hat's a good point that access() needs to be used instead of stat() (well, for some implementations of ACLs, I guess).

So, why isn't this fix needed for every possible file test? Shouldn't that pragma be in some "BEGIN{}" block or something?

[tschoonj]

If there is a way for the Demeter executables to start with this pragma enabled, and that this would apply to all modules they use, then that would clearly be the best solution.

@bruceravel I would appreciate your thoughts here. If you could give me some ideas, I would gladly open a PR that would solve this once and for all.