Public Enum FileAttributes is missing values

nzstevem commented 4 years ago

Description

The System.IO.FileAttributes Enum is missing values associated with OneDrive (and iCloud) files not present on the local system.

The following values from

C:\Program Files (x86)\Windows Kits\10\Include\10.0.16299.0\um\winnt.h:

are not present.

FILE_ATTRIBUTE_UNPINNED = 0x00100000 FILE_ATTRIBUTE_PINNED = 0x00080000 FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS = 0x00400000

Enum.GetName returns null for these file attribute flag values.

Configuration

Win10 19041.388, VS 16.7.0 PV 5, NetCore 5.0.100-preview.6.20318.15

Other information

I can't find any official docs defining the function of these bits but I can confirm they are being used by iCloud and I assume OneDrive. A file not present on a local system has the unpinned and recall bits set. Once downloaded (automatically when accessed) these are cleared. It would useful to be able to programmatically force immediate "space saving" on individual files.

KalleOlaviNiemitalo commented 4 years ago

FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS and FILE_ATTRIBUTE_RECALL_ON_OPEN are documented in File Attribute Constants.

FILE_ATTRIBUTE_PINNED and FILE_ATTRIBUTE_UNPINNED are mentioned in FILE_RENAME_INFORMATION structure. However, I wonder if applications should call CfGetPlaceholderInfo and CfSetPinState instead of getting and setting those file attributes directly.

nzstevem commented 4 years ago

A few more thoughts...

There's a need here for a generally more sophisticated api. I've been working with the iCloud photo directory. Apple have recently changed the way this works - by default only links are downloaded by the win10 icloud client, the file names have become GUID like (useless for a human to work with), and they no longer support down loading to a directory hierarchy so you end up with 1000's of files in one directory. As these directories get larger (mine is already over 7000 files) performance will get progressively worse.

I assume that the onedrive and icloud clients will become multi platform. An api will ideally anticipate and abstract the underlying differences.

Just as a start, from managed code there needs to be obvious ways to a) identify the datetime of the last directory sync, b) force a directory sync with an async option, c) request a file download async, d) receive download progress notification (think about a large video file ), e) remove a file from the client without deleting it back up in the cloud, f) trap the specific error that the network is not connected (so the user can be prompted to remedy the issue) rather than just fail (The cloud operation is invalid.) which happens from time to time even when the network connectivity is good.

danmoseley commented 4 years ago

Would you consider updating the top post to be in the format of API Proposal ?

Generally we want to only represent concepts that make sense across Windows, macOS and Linux, so some thoughts about what that would mean would be good.

nzstevem commented 4 years ago

I think there are two issues. One easily fixed - the exposure of the attribute values in the Enum. Hold this issue open for that alone.

And then the much larger issue of how to provide the required functionality in software that is specifically written to anticipate and be friendly to these environments. Even a simple thing like a default file open dialog needs a progress bar, a cancel option, and a non-fatal response if the file can't be retrieved. And then there is the issue of being allowed or not to consume metered bandwidth. And how to handle local storage size limitations. And... Probably best to start a new train of thought in a new issue. Let me dwell on this for a few days.

HEIC-to-JPEG-Dev commented 1 year ago

There are more issues than exposing the new enums. Windows decides that some apps are in compatibility mode for file attributes and hides the real bits. It often returns attributes that are for a normal file, when in fact the file is part of a cloud sync and currently local. I’d love you use the new Windows.Storage namespace, but it has the performance of a one legged dog, going up stairs, while being tied down.

danielniccoli commented 11 months ago

This bug blocks a PowerShell cmdlet Copy-Item from working.

A file that was on OneDrive at any point in time will have a one of these flags set permanently. When trying to copy that file to a remote server via PowerShell, it will fail with the following error:

Line |
   5 |      Copy-Item .\lgpo.txt C:\lgpo\ -ToSession $s
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception setting "Attributes": "Cannot convert value "524320" to type "System.IO.FileAttributes" due to enumeration values that are not valid. Specify one of the following enumeration values and try again. The possible enumeration values are "ReadOnly;Hidden;System;Directory;Archive;Device;Normal;Temporary;SparseFile;ReparsePoint;Compressed;Offline;NotContentIndexed;Encrypted;IntegrityStream;NoScrubData"."

In the above case the file lgpo.txt was synced to OneDrive, but then moved to C:\temp\lgpo.txt. It has some flags permanently set. The file was then copied to a PSSession with the command Copy-Item .\lgpo.txt C:\lgpo\ -ToSession $s. The PSSession was opened to a Windows Server 2019.

danmoseley commented 11 months ago

@SteveL-MSFT is there any way in Powershell script to work around the above case? In C# one can simply cast any value to the enum . Is it possible for Powershell to allow that? We cannot generally add new API such as enum members in servicing.

KalleOlaviNiemitalo commented 11 months ago

In PowerShell script, one can call [Enum]::ToObject([System.IO.FileAttributes], 0x99999999) to cast any value to the enum type. However, I don't think the user can inject such a cast into how Copy-Item does remoting. It would have to be fixed in PowerShell, perhaps around here:

https://github.com/PowerShell/PowerShell/blob/v7.3.8/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L4285-L4308 attributes from metadata to local file
https://github.com/PowerShell/PowerShell/blob/v7.3.8/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L4808-L4809 attributes from local file to metadata
https://github.com/PowerShell/PowerShell/blob/v7.3.8/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L9544-L9549 attributes from remote file to metadata
https://github.com/PowerShell/PowerShell/blob/v7.3.8/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L9050-L9054 attributes from metadata to remote file; I think this is what blows up

But even if the unrecognized file attributes survived PowerShell remoting, I think they might still be rejected by .NET on the remote computer, when copying from Windows to Unix.

https://github.com/dotnet/runtime/blob/7db4f33f5b00ac9f5e65dd690b3f69969cb24e63/src/libraries/System.Private.CoreLib/src/System/IO/FileStatus.Unix.cs#L226-L239

jeffhandley commented 10 months ago

To pursue this, we'll need to conduct research across all platforms and cloud file sync systems.

HEIC-to-JPEG-Dev commented 1 day ago

It should be noted that MacOS and Windows have the same presentation of files in this way - e.g. they both have hte concept of placeholder files that are local, in the cloud, or pinned. It should also be noted that .Net doesn't like placeholder files that are local (but I think this is an OS issue).

dotnet / runtime