ls -l causes OneDrive to sync files from server.

[titusjan]

Describe the issue

When I use the ls -l command in a directory that is under control of OneDrive, it will download the file from the cloud. If I don't use the -l option, the file is not downloaded. This makes ls -l dangerous (or at least annoying) to use because there is always the risk of downloading a large number of files if you accidentally perform an ls -l in a OneDrive directory.

In Cygwin using ls -l does not show this behavior.

Steps to Reproduce the Problem

1. Open the Windows Explorer and go to a directory that is under control of the OneDrive
2. Right click on the files and select "Free up space"
3. After this, all the files should have a blue-white cloud icon, which indicates that they are stored only in the cloud.
4. Open an MSys2 bash shell and cd to that same directory
5. Type ls
6. The icons in the explorer do not change. This is desired behavior (see the first screen shot)
7. Type ls -l
8. The OneDrive will download all files from the cloud before printing them in the ls output. This is undesired behavior. All files now have a green check mark icon, which indicates that a local version has been stored on the machine. (see the second screen shot)

Additional info

MSYS2 coreutils 8.32-2 Cygwin coreutils 8.32-1 Windows 10 Home: 19043.1766 OneDrive for Windows - Version 22.111.0522.0002 (June 3, 2022)

Question

So can you please fix it, or provide a work-around? If not, could you please explain what's going on?

I suspect that ls -l opens the files whereas just ls does not.

Like I said, in Cygwin ls -l performs correctly so I think the issue lies with MSys2. That's why I report it here.

[codesmacgodes]

"find" and "stat" have the same problem with respect to OneDrive, and like "ls", "find" and "stat"s corresponding cygwin implementations do not have this problem. I think this is further evidence that the problem lies with MSYS2 and that it is correctable there, rather than in ls, find, and stat, etc. etc.

[mwisnicki]

See the thread https://sourceware.org/pipermail/cygwin/2024-March/255613.html Duplicated in https://github.com/msys2/msys2-runtime/issues/206

[mwisnicki]

I didn't check cygwin, interesting that it seems to work in it.

[codesmacgodes]

I think it must be because cygwin will actually let you use chmod to set permissions on files, roughly equal to what you'd get in a unix-like environment. It's not a perfect translation, but it does it (and the windows permissions dialog will complain about them being entered in the wrong order).

[BrianInglis]

Cygwin does its best to emulate NFS V4 ACLs IIRC? and conform to POSIX requirements within MS Windows ACL limitations. Note that nothing will complain about Windows ACL permissions, except if you attempt to modify the ACLs in MS Windows File Explorer, which will complain if they are not in MS Windows File Explorer's "expected" NTFS ACL order, which can also happen from using MS Windows ACL commands - of course this is not defined or documented except SpiceWorks How to correctly order NTFS ACL using icacls refers to an HPE article saying: "Windows 2000 introduced a new inheritance model in which directly applied access control entries (ACEs) have precedence over inherited ACEs. The computer implements this precedence by placing directly applied ACEs ahead of inherited ACEs in a discretionary access control list (DACL)" and this seems to agree with Cygwin's ACL order and MS Windows preferred Order of ACEs in a DACL so it may just be a limitation of MS Windows File Explorer's ACL handling.

[mwisnicki]

As far as I can see Cygwin has the same file reading logic as MSYS: https://www.cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/fhandler/disk_file.cc#n536 but I can confirm behavior is different:

MSYS (Git Bash):

$ ls -l .ssh/known_hosts
-rw-r--r-- 1 mwisn 197609 4885 Mar 18  2023 known_hosts
$ mount
C on /c type ntfs (binary,noacl,posix=0,user,noumount,auto)

Cygwin:

$ ls -l .ssh/known_hosts
-rwx------+ 1 mwisn mwisn 4885 Mar 18  2023 known_hosts
$ getfacl /cygdrive/c/Users/mwisn/.ssh/known_hosts
# file: /cygdrive/c/Users/mwisn/.ssh/known_hosts
# owner: mwisn
# group: mwisn
user::rwx
group::---
group:SYSTEM:rwx        #effective:---
group:Administrators:rwx        #effective:---
mask::---
other::---
$ mount
C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)

Looks like cygwin is using different attribute mapping.

If I mount with acl in MSYS it will stop the reads:

$ mount -o binary,acl,posix=0,user,auto C: /c1
$ ls -l /c1/Users/mwisn/.ssh/known_hosts
-rwx------+ 1 mwisn 197609 4885 Mar 18  2023 known_hosts*

Of course it will make all the files executable since that's the default ACL in Windows.

[mwisnicki]

I tired test version of cygwin and it fixes the problem in noacl case in cygwin.

See: https://sourceware.org/pipermail/cygwin/2024-March/255642.html

Looking forward to g06aa5a751682 making it's way to MSYS!

[titusjan]

Thank you very much for picking this up.

Looking forward to g06aa5a751682 making it's way to MSYS!

Can you give a heads up when it does?

[eidsnessje-nih]

Found the spot in MSYS2 where it opens and reads the file. I haven't been able to put in the time and fully understand the cygwin & the buildXL situation that has been linked, but I think the prediction is correct that it will delay the opening (and downloading) of files from the ntOpenFile until the ntReadFile. It wants its 3 bytes, and I don't suppose there's any easier way to get them.

https://github.com/msys2/msys2-runtime/blob/44b5e4eed0f2f119d312fe718c2124250c8626f9/winsup/cygwin/fhandler/disk_file.cc#L526

All that said, it's exciting news to me that we can remount /c/ with acl or noacl. It's not often that downloading OneDrive files is a huge problem, so having a workaround for those situations that allows me to continue with "Git Bash" is just fantastic. It just didn't occur to me that we had that available to us.

Thanks everybody!

Just as an aside, the same underlying behavior affects speed of directory listings on network drives, too, but that would be a separate issue, and I'm not sure it's so easy to work around in such cases. I don't have reason to believe ACLs would be any easier to work with than the first 3 bytes of each file, and I don't have any ideas. My brief experimentation with that is not fully informed, and I just made things slower for myself (and then put it back the way I found it).

[mwisnicki]

The fix for this was already merged to MSYS2: https://github.com/msys2/msys2-runtime/pull/209