carlreinke commented 6 years ago

Edit by @carlossanlop: Revisited API Proposal

Original proposal:

Rationale

The ability to interact with symbolic links (symlinks) in .NET is currently limited to determining that a file is ReparsePoint. This proposed API provides the ability to identify, read, and create symbolic links.

Proposed API

public class Directory
{
    public static DirectoryInfo CreateSymbolicLink(string linkPath, string targetPath);
    public static string GetSymbolicLinkTargetPath(string linkPath);
    public static bool IsSymbolicLink(string path);
}

public class File
{
    public static FileInfo CreateSymbolicLink(string linkPath, string targetPath);
    public static string GetSymbolicLinkTargetPath(string linkPath);
    public static bool IsSymbolicLink(string path);
}

public class FileSystemInfo
{
    public bool IsSymbolicLink { get; }
    public string SymbolicLinkTargetPath { get; }

    public void CreateSymbolicLink(string linkPath);
}

Details

The path returned from GetSymbolicLinkTargetPath(string)/SymbolicLinkTargetPath will be returned exactly as it is stored in the symbolic link. It may reference a non-existent file or directory.

For the purposes of this API, NTFS Junction Points are considered to be like Linux bind mounts and are not considered to be symbolic links.

Updates

Move GetSymbolicLinkTargetPath and IsSymbolicLink from Path to Directory, DirectoryInfo, File and FileInfo.
Add CreateSymbolicLink.
Split from dotnet/runtime#23871.
Change path to linkPath where a link file's path is desired.
Move the APIs to the FileSystemInfo base class.

JeremyKuhne commented 6 years ago

Adding support for symbolic links is particularly important now that Windows has changed the default permissions for creating them. (In the Fall Creator's Update you no longer need to be elevated to create them)

MSDN Symbolic Links

One of my concerns is how we introduce support for junctions, mounts, or possibly Mac aliases. Would we want to be more generic here, like GetLinkType(path)? There is an immediate need to filter both Junctions and Symbolic links from other reparse point types on Windows. I'm also wondering if we really want to add this to all of the classes, or just Path. I'm not sure if "link to me" (*Info classes) is that useful?

namespace System.IO
{
    public static class Path
    {
        // Or perhaps GetKnownLinkType()?
        public static LinkType GetLinkType(string path);
        public void CreateSymbolicLink(string linkPath, string targetPath);
    }

    public enum LinkType
    {
         None,
         Symbolic,
         Junction
    }
}

carlreinke commented 6 years ago

As far as I know, symbolic links are the only link type that is available on all platforms that .NET Core targets, which makes them more useful than other link types. I don't see a lot of value in supporting other link types, but perhaps I'm just being short-sighted.

Windows differentiates between file and directory symbolic links, so you can't just have Path.CreateSymbolicLink(string, string). (The target doesn't need to exist, so you couldn't query the target to see what kind of symbolic link to create.)

The *Info methods are for creating a link at the path of the *Info to a given target path, so "link from me". (Similar to *Info.Create().)

JeremyKuhne commented 6 years ago

I don't see a lot of value in supporting other link types, but perhaps I'm just being short-sighted.

I'm not expecting creation to become a priority, but identifying Junctions is important (over all the other reparse points, those two need special treatment when enumerating).

The target doesn't need to exist

I didn't even realize that. Is that case useful to support?

The Info methods are for creating a link at the path of the Info to a given target path, so "link from me".

I would think "link to" makes more sense. Creating a FileInfo for the target seems backwards from the way I'd expect people to come at this.

carlreinke commented 6 years ago

The target doesn't need to exist

I didn't even realize that. Is that case useful to support?

If I'm creating an arbitrary directory structure with some links in it, I wouldn't want to have to figure out what order I need to create the file and directories in order to make it work. Unzipping a .zip file with symbolic links in it, for example. (That's hypothetical at this point though — .zip probably doesn't store whether the symlink is to a directory or a file.)

I would think "link to" makes more sense. Creating a FileInfo for the target seems backwards from the way I'd expect people to come at this.

Maybe I'm confused. I would expect that:

File.CreateSymbolicLink(string, string) would return a FileInfo for the symbolic link file that was created.
FileInfo.CreateSymbolicLink(string) would create a symbolic link file at FileInfo.FullName that targets the path provided in the argument.

omariom commented 6 years ago

How does PowerShell handle links?

JeremyKuhne commented 6 years ago

@carlreinke Sorry I haven't responded, I've been out on an extended vacation. @jhudsoncedaron is also interested in this area and has opened https://github.com/dotnet/runtime/issues/24655. We should leverage the interest here and try and get a strong proposal together that we can easily clear through API review.

If I'm creating an arbitrary directory structure with some links in it, I wouldn't want to have to figure out what order I need to create the file and directories in order to make it work.

Agreed, thanks for the example.

Maybe I'm confused.

Sorry, terminology is a little weird and I think I was confusing myself. "Create a symbolic link to me" is what I meant, which is what you're describing I think.

jhudsoncedaron commented 6 years ago

I hate to disappoint you guys but "create a directory structure with some links in it" is a pill on Windows because Windows decided to demand knowing whether the target is a file or a directory at create time.

JeremyKuhne commented 6 years ago

"create a directory structure with some links in it" is a pill on Windows because Windows decided to demand knowing whether the target is a file or a directory at create time.

Yeah, not sure why that is- I'll take a look and see if I can find any clues, but I suspect it has some compelling legacy reason that no longer applies.

JeremyKuhne commented 6 years ago

I'm marking this one ready for review with one tweak:

public class DirectoryInfo
{
    // Create a symbolic link at the given path to this directory info
    public void CreateSymbolicLink( string path );
}

public class FileInfo
{
    // Create a symbolic link at the given path to this file info
    public void CreateSymbolicLink( string path );
}

Info classes can be created for non-existent paths. We will allow creating symbolic links at the given path regardless of the info existence if the OS/FileSystem allows it.

carlreinke commented 6 years ago

It seems to me that it's inconsistent if

FileInfo.Create() creates a file at FileInfo.FullPath and
FileInfo.CreateSymbolicLink( string ) does not create a (symlink) file at FileInfo.FullPath.

(And similarly for DirectoryInfo.)

jhudsoncedaron commented 6 years ago

@carlreinke : There's a bunch of other problems with this API scheme leading me to have to abandon hope of correcting it.

JeremyKuhne commented 6 years ago

It seems to me that it's inconsistent

@carlreinke I don't think it's a problem, but I'll bring it up at the review. I think it is more of an issue to have IsSymbolicLink change values.

There's a bunch of other problems with this API scheme

@jhudsoncedaron Can you please articulate issues with these specific additional API's here so we can consider them.

I spoke with @pjanotti and we believe that we should be using linkPath and targetPath for additional clarity.

I'm updating the main comment to reflect the discussion.

jhudsoncedaron commented 6 years ago

@JeremyKuhne : It expects the caller to follow links one at a time. To work correctly, the caller must pass an argument to the constructor that specifies whether to get information about the link or the target of the link. It is context sensitive which one the caller would want.

Also, the results of getting one needs to know if its some strange type of file node or not. We probably don't need to handle fifos, block specials, etc. but we at least need to allow directory walking algorithms to know if they encountered one so they can skip over it.

JeremyKuhne commented 6 years ago

It expects the caller to follow links one at a time.

I'm fine with public static string GetSymbolicLinkTargetPath(string linkPath, bool recurse = false); We'd have to track cycles and should probably throw.

To work correctly, the caller must pass an argument to the constructor that specifies whether to get information about the link or the target of the link.

I don't see why. We currently don't have a constructor that says to follow to the end so they're always information about the given path. You create a file, you have an Info on the file. You create a link, you get an Info to the new link entry. Nothing should have to change.

That said I think it is probably worth adding a convenience constructor that follows links to the final path. Not having it initially, however, shouldn't block this from moving forward. The biggest issue right now is that we have no way to deal with links. Getting the basics in place for the next release to unblock people is super important I think. That window is rapidly closing.

jhudsoncedaron commented 6 years ago

That said I think it is probably worth adding a convenience constructor that follows links to the final path.

How many times do I have to tell you that won't work? Recursively reading the link is not the same as asking the OS for the link target information.

It's the difference between lstat() and stat(). I already provided the fragment for constructing stat()'s behavior on Windows. Hint: you don't know how many times the OS recurses until it gives up or if readlink() even returned something that can be followed.

We currently don't have a constructor that says to follow to the end so they're always information about the given path

So add one.

JeremyKuhne commented 6 years ago

Recursively reading the link is not the same as asking the OS for the link target information.

Could you please provide supporting links and/or some code to clarify? I don't see why recursively getting target paths for symbolic links wouldn't give you an actual file path and I'm struggling to find supporting material.

I'll continue to spend time in the near term investigating the various APIs and looking at the implementation internals.

jhudsoncedaron commented 6 years ago

new FileInfo("/dev/stdin");

This file really does exist on *nix systems and can be opened if you have a handle to it. But it can't be resolved by readlink() recursively.

 ~$ cat | ls -l /dev/stdin
 lrwxrwxrwx 1 root root 15 Oct  9 08:18 /dev/stdin -> /proc/self/fd/0
 ^C
 ~$ cat | ls -l /proc/self/fd/0
 lr-x------ 1 DOMAIN\jhudson DOMAIN\domain users 64 Jan 18 13:01 /proc/self/fd/0 -> pipe:[2166247]
 ^C
 ~$ cat | ls -l /proc/self/fd/pipe*
 ls: cannot access /proc/self/fd/pipe*: No such file or directory
 ^C
 ~$ cat /dev/stdin
 Hi
 Hi
 ~$

This is not the only example, just the easiest found. When you consider that your recursive symbolic link descent doesn't match the kernel's and when you throw in things like sshfs it's quickly best to go ahead and just assume that readlink() is for informational use only.

JeremyKuhne commented 6 years ago

This is not the only example, just the easiest found.

Thanks, I'll play around with this a bit more and respond to the thread.

JeremyKuhne commented 6 years ago

@jhudsoncedaron

Why does one get pipe:[...] when piping through cat? While I can replicate your example I also get the following:

~$ ls -l /dev/stdin
lrwxrwxrwx 1 root root 15 Jan 18 15:15 /dev/stdin -> /proc/self/fd/0
~$ ls -l /proc/self/fd/0
lrwx------ 1 jeremy jeremy 0 Jan 18 15:29 /proc/self/fd/0 -> /dev/tty1
~$ ls -l /dev/tty1
crw-rw---- 1 jeremy tty 4, 1 Jan 18 15:15 /dev/tty1
~$ readlink /dev/stdin
/proc/self/fd/0
~$ readlink /proc/self/fd/0
/dev/tty1
~$ readlink /dev/tty1
~$ readlink -f /dev/stdin
/dev/tty1

Working with any of the intermediate paths seems to work fine.

jhudsoncedaron commented 6 years ago

Because the link is actually to the file by direct reference, and the path displayed is merely the path used to open the file when it was opened (i.e. potentially stale (file moved or deleted out from under you), wrong namespace (chroot() or something more exotic -- note the whole system except for a couple of process is in a chroot() jail these days), or completely irrelevant (pipe()).

Paths in /proc are often passed from process to process to prevent somebody getting way too clever and finding a way to hijack them. This only works because of the fact that /proc symbolic links are directly resolved. This results in the file that was intended even if somebody else renames it out of the way. Therefore, if the API tries to do readlink() itself rather than the direct resolution the security principle is disabled.

We also note that any userspace virtual filesystem can choose to do this.

JeremyKuhne commented 6 years ago

I don't understand how your example shows that walking can't be done. Running readlink(0) is effectively the same as running readlink(1) in this case, right? If I walk these with readlink I get a usable, direct path at the end.

jhudsoncedaron commented 6 years ago

One of the tests shows that /proc/self/fd/pipe:[2166247] doesn't exist. Its most-resolved name is /proc/self/fd/0 which can be opened despite being a symbolic link to a path that doesn't exist.

Also, consider the following C#:

    using (var f = new FileStream("/tmp/zxqxvm", FileMode.Create, FileAccess.ReadWrite))
    {
          File.Delete("/tmp/zxqxvm");
          using (var fv2 = new FileStream("/proc/self/fd/" + f.SafeFileHandle.DangerousGetHandle().ToString()))
          {
                /* succeeds */
                if (!new FileInfo("/proc/self/fd/" + f.SafeFileHandle.DangerousGetHandle().ToString()).Target.Exists)
                        throw new Exception("I opened a file but it doesn't exist.");
          }
    }

If this were written as a test case it should pass.

JeremyKuhne commented 6 years ago

One of the tests shows that /proc/self/fd/pipe:[2166247] doesn't exist. Its most-resolved name is /proc/self/fd/0 which can be opened despite being a symbolic link to a path that doesn't exist.

But where would /proc/self/fd/pipe:[2166247] enter into this API? We'd never give that back, we'd give back /dev/tty1.

If this were written as a test case it should pass.

In the other proposal .Target is meant to open the target of the symlink, which has been deleted. On Windows it will be true, but only because the file handle is still open. As soon as it the handle closed, it goes false. That said, there is definitely a big difference in behavior here. Windows won't let you open the symlink if the target file is marked for deletion. You get access denied. Take the following directory layout and sample code:

01/18/2018  01:07 PM                 4 a
01/18/2018  01:07 PM    <SYMLINK>      b [a]
01/18/2018  01:08 PM    <SYMLINK>      c [b]
01/18/2018  07:30 PM    <SYMLINK>      d [e]

FileInfo fi = new FileInfo(@"F:\test\links\e");
using (var tempfile = new FileStream(fi.FullName, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.Delete | FileShare.ReadWrite))
{
    File.Delete(fi.FullName);
    using (var fs = new FileStream(@"F:\test\links\a", FileMode.Open, FileAccess.Read)) { }
    using (var fs = new FileStream(@"F:\test\links\b", FileMode.Open, FileAccess.Read)) { }
    using (var fs = new FileStream(@"F:\test\links\c", FileMode.Open, FileAccess.Read)) { }

    // Target still exists, but can't be deleted
    fi.Refresh();
    Console.WriteLine($"File {(fi.Exists ? "exists" : "doesn't exist")}.");

    // Access denied if the file is deleted and all of the handles aren't closed
    // using (var fs = new FileStream(@"F:\test\links\d", FileMode.Open, FileAccess.Read)) { }
}

// Target will no longer exist
fi.Refresh();
Console.WriteLine($"File {(fi.Exists ? "exists" : "doesn't exist")}.");

I don't think that you can create a handle on a file marked for deletion on Windows. I might be remembering incorrectly, however. It is an interesting test case for link handling- I'll check how GetFinalPathNameByHandle/etc. deal with a handle opened on the link itself.

jhudsoncedaron commented 6 years ago

But where would /proc/self/fd/pipe:[2166247] enter into this API? We'd never give that back, we'd give back /dev/tty1.

You must be having trouble reproducing. In this case, stdin was a pipe handle where cat was piped to ls. Try cat | readlink -f /dev/stdin

Windows won't let you open the symlink if the target file is marked for deletion.

That's a property of NTFS. If you fix your share modes to include FileShareDelete in all those opens and try it with a UNC path to a set-top NAS box it will work (or the path might go away immediately leaving you unable to open it).

The general point being is readlink() is for informational use only. If you want to get the information about a link target, use the API to get information about a link target (stat() or GetFileInformationByHandle(). This is guaranteed to resolve the link the exact same way that opening the link path would. If there's no .NET API to get information about the linked target by the underlying native API call but only resolving by its own readlink() calls I can construct case after case where the .NET API generates spurious failures on static filesystems.

JeremyKuhne commented 6 years ago

Windows won't let you open the symlink if the target file is marked for deletion.

That's a property of NTFS.

It may be, but it is important. If we're suggesting you can find the state of the link target by opening a handle on it and you can't- we'll have to jump through some serious hoops. We already use FindFirstFile as a workaround for this issue in our code. I'm going to try to see if I can use GetFinalPathNameByHandle to implement the same hack- I'll respond back to the thread when I've finished looking at it.

In any case, we can't rely on leaning too heavy on the link info as the only way to get it programmatically on Windows is through DeviceIoControl, which won't fly for WinRT at this point. I still want to expose the data where we can though.

Providing access to file info through handles has been on my bucket list for some time. Providing a static method on FileSystemInfo that will construct a file/directory info is probably doable FileSystemInfo.CreateFromHandle(SafeFileHandle handle), but won't handle the deleted scenario.

Here is what I'm currently thinking:

public class FileSystemInfo
{
    public bool IsSymbolicLink { get; }

    // The string value of the target for symbolic links
    public string SymbolicLinkTarget { get; }

    public void CreateSymbolicLink(string linkPath);
    public static FileSystemInfo CreateFromHandle(SafeFileHandle handle);
}

public class FileInfo
{
    public FileInfo(string fileName, FileSystemInfoFlags flags);

    // this pointer or FileInfo on the target if this info is a SymbolicLink
    public FileInfo TargetInfo { get; }
}

public class DirectoryInfo
{
    public DirectoryInfo(string fileName, FileSystemInfoFlags flags);

    // this pointer or DirectoryInfo on the target if this info is a SymbolicLink
    public DirectoryInfo TargetInfo { get; }
}

[Flags]
public enum FileSystemInfoFlags
{
    // The class will contain information about the final target of symbolic links
    FollowSymbolicLinks = 0x1;

    // Other options in future 
}

jhudsoncedaron commented 6 years ago

If we're suggesting you can find the state of the link target by opening a handle on it and you can't- we'll have to jump through some serious hoops.

(assuming you mean because it has a pending delete) I was planning on having FileInfo return the status of "you don't have permission to resolve the link target" in this case, same as if the link target referred to a directory you can't traverse. I already use this technique on Windows native code to see through links.

Oh that reminds me. The behavior of new FileInfo("C:\\DirectoryYouHaveNoAccessTo\\somefile.txt") is not documented yet.

public FileInfo(string fileName, FileSystemInfoFlags flags);

Looks good to me.

jhudsoncedaron commented 6 years ago

So on checking, my intention to follow the behavior for no permissions to directory for a link to a deleted file is actually great.

If you're checking if the file exists because you want to prompt for overwrite (new FileInfo(filename, 0).Exists (or I think File.Exists(filename) does the job). This treats a dangling symbolic link and a link to a file with a pending delete as exists, which is probably what you want.

If you want to do a file type decision ladder (is this a file or directory I was passed), (new FileInfo(filename, FileSystemInfoFlags.FollowSymbolicLinks).Attributes is right barring exotic cases such as it's a persistent fifo. A symbolic link to a file with a pending delete and a dangling symbolic link will both report 0 for attributes resulting in the decision ladder performing no action (since you can't open it this is what you want). Right now, the fallback of calling FindFirstFileEx causes something silly to happen if it does have a pending delete but oh well. The "file" side of the decision will just have to deal with it like it would have to deal with the file disappearing in between the two calls.

If you want to perform a file open operation just open the file first, then if you don't pass FileShare.Delete you don't have to worry about the zombie state in the first place.

stormCup commented 4 years ago

Imho the file api should always work on the target of a symlink (except for delete).

jhudsoncedaron commented 4 years ago

@stormCup : We've been through this already. Sometimes you want to know about the symlink target; other times you want to know about the symlink. It needs a flags for new FileSystemInfo.

stormCup commented 4 years ago

Yes, I agree there needs to be an api that allows acces to both. What I’m saying is hat the old app should access the target. I understand this is a breaking change, but I’d argue that any code written without knowledge of the new api that’s accessing symlinks is broken anyway: it never makes sense to read the contents of a symlink using a legacy file api.

carlossanlop commented 4 years ago

Triage: We should include hard link support as discussed in issue https://github.com/dotnet/corefx/issues/29978

kdbotts commented 4 years ago

Attached is an implementation of reading symbolic links in .NET under Windows. (Essentially, an emulation of readlink() from *ix.) Figuring out the driver level bit-fiddling took somee effort, but it has been working well for several years now. Do whatever you like with it: I ask no credit or any such.

Curiously, I now find myself deploying via DotNetCore to Linux, and I have to figure out how to P/Invoke to the real readlink(). A common API actually in the DotNetCoreAPI would be mighty nice... WinReparsePoint.cs.txt

jhudsoncedaron commented 4 years ago

@kdbotts : I put this on nuget and github months ago: https://www.nuget.org/packages/Emet.FileSystems/0.0.1

mklement0 commented 4 years ago

@omariom, re how PowerShell handles links:

It uses its ETS (Extended Type System) to decorate FileInfo / DirectoryInfo instances with additional properties that call public engine methods:

System.String LinkType { get=GetLinkType; }
System.String Target { get=GetTarget; }

.LinkType is one of the following strings: SymbolicLink, HardLink and, on Windows only, Junction and AppExeCLink
Target is the symlink / junction's target path string, as defined in the file-system item at hand; that is, it may be a relative path.

Both properties return $null for non-symlinks/non-reparse points.

There's a pending proposal to add a .ResolvedTarget property that returns the full, ultimate target path in canonical form, possibly wrapped in a FileInfo / DirectoryInfo instance - see https://github.com/PowerShell/PowerShell/issues/13366

This, and the related proposal to implement Convert-Path -Canonical (https://github.com/PowerShell/PowerShell/issues/10640) brings me to suggest bringing this functionality directly to .NET:

I propose introducing an additional System.IO.Path.GetFullPath() overload that resolves any path to its canonical form, irrespective of whether the path is itself a symlink/reparse point, or whether symlink/reparse points are among the ancestral path components only, or neither:

public static string GetFullPath(string path, FileSystemInfoFlags flags);

If flag FollowSymbolicLinks is passed:

For a path that contains no symlink/reparse-point components, it would act like GetFullPath() currently does.
Otherwise, symlinks/reparse-point path components would be resolved recursively to get the ultimate target file/directory's path in full, normalized form.

The details of the discussion above are beyond me, but I think the following APIs are relevant:

Windows: The GetFinalPathNameByHandle Windows API function.
Unix: The POSIX-mandated realpath() function (implemented as library functions on macOS and Linux; see man 3 realpath).

The desired behavior with respect to the existence of the input path and its target could be handled with additional FileSystemInfoFlags enum values.

Since GetFullPath() currently makes no assumptions regarding existence (e.g., System.IO.Path.GetFullPath("/totally/../made/up") happily returns "/made/up"), this should probably be the default behavior if only flag FollowSymbolicLinks is passed.

In order to require that the input path and/or its target exist, additional flags could mimic the GNU readlink utility's -e / -f / -m options.

LarinLive commented 3 years ago

Hello. Could you clarify the status of this task? I'm fed up with importing GetFinalPathNameByHandle function. Can we hope that the symbolic link API will have been delivered in the near future?

Joe4evr commented 3 years ago

Even if there was an update on this issue, it hasn't been reviewed and approved by the Framework team. And .NET 5 is going to release tomorrow, so you're gonna be waiting for .NET 6 next year at the earliest.

@JeremyKuhne Is there anything missing keeping this from going to API review?

iSazonov commented 3 years ago

Please vote with 👍 on OP.

jhudsoncedaron commented 3 years ago

@AntonPlotnikov : Do you want ReadSymbolicLinkRecursive()? I have a library and could add such a function.

ericstj commented 3 years ago

In creating a workaround for https://github.com/dotnet/runtime/issues/36091 I prototyped some of this functionality, learning from the powershell implementation and docs. https://github.com/ericstj/sample-code/tree/runtime36091/symLinkConfig

I think someone just needs to pick this up where @JeremyKuhne left off and drive those APIs through review. I'm not seeing anything blocking in the discussion above.

jhudsoncedaron commented 3 years ago

@ericstj: Don't bother. The review team will not do their job.

zivkan commented 3 years ago

The NuGet team has an issue at the moment where the dotnet CLI and nuget.exe via mono read different user-profile nuget.config files. One idea to resolve this is to set up a symlink from one location to the other. Therefore, having BCL APIs for us to use avoids us needing to P/Invoke and creating our own internal APIs, and all the cross platform testing that involves.

I'm unsure on the timing of when we'll implement this, but what's the procedure to get this API as a partner-ask?

jhudsoncedaron commented 3 years ago

@zivkan : I tore the proposed API to shreds already. It should fail API review not pass it.

zivkan commented 3 years ago

Then someone needs to take your feedback into account and propose new APIs.

jhudsoncedaron commented 3 years ago

@zivkan : I published a working library on Nuget; thought I don't have a Mac to test it on. APIs aren't copyrightable so help yourself to the API surface area.

terrajobst commented 3 years ago

@jhudsoncedaron

@ericstj: Don't bother. The review team will not do their job.

I'm surprised by the harshness of this comment. What did we do (or didn't do)?

jhudsoncedaron commented 3 years ago

@terrajobst : Didn't do is the case. 1. This item left for three years. 2. Conflict between "do not throw in dispose" and "need to tell if FileStream.Dispose()" ran out of disk flushing the OS buffer was finally resolved by giving up on waiting for the API review team to answer. I've paid too high a cost by guessing wrong which way to go after multi-year delays.

terrajobst commented 3 years ago

Apologies. I understand that this can be frustrating. I hope you can also see that it's not like we're sitting idle here. We have over 5,000 open issues and frequently have over 200 open PRs . We do our best to work through the backlog but the reality with working on popular OSS projects is that at any given moment there are more things that could be done than you have resources for. I won't claim that we always prioritize/triage correctly, but the bottom line is that we must be selective in order to get anything done.

The same is true for the API reviews themselves. The team performing these meets every week for at least two hours. And in many cases multiple times in order to accommodate spikes. All reviews are recorded & streamed, in case you want to see what this is like. In this particular case, the API never reached the stage where the feature owner deemed it ready for review, which is why we never ended up reviewing this issue. Is this ideal? Of course not. But again there are over 700 open issues in this category. We try hard to add new APIs but at the same time bugs or work to support cross-cutting features across the .NET stack tend to get priority.

I'll see what we can do here; I think sym links have popped up often enough now that it seems like something that we might want to prioritize.

jeffhandley commented 3 years ago

Hi folks. I was no following the comments on this issue closely; sorry for not chiming in sooner. We have this issue and several others related to file links accumulated on a project board. Later during the 6.0 release cycle, we have some time earmarked to look at the collection of issues there holistically and determine what investments we can make in the 6.0 release. New APIs in this area would likely be targeting Preview 5 and/or Preview 6.

iSazonov commented 3 years ago

Open questions for me:

Should we consider absolute/relative path for symlink targets?
Should we take into account a target existence? Has TryCreateSymbolicLink()? Or a additional flag in CreateSymbolicLink()?

As I commented in https://github.com/dotnet/runtime/pull/47348#issuecomment-785635745 I believe we need FileExtendedAttributes property. Perhaps this could force to change the proposal too.

See how rich Windows reparse points are - simple reparse point, surrogates, named and non-named. They cover hardlinks, symlinks, mount points, OneDrive, Appx (something else?). If we look MacOS (BSD) file extended attributes, perhaps we need more general API then the proposal.

jhudsoncedaron commented 3 years ago

So now I have something to say. I did make attempts to design an API that fit into the existing FileInfo/DirectoryInfo stuff. File.CreateSymbolicLink(string, string) and Directory.CreateSymbolicLink(string, string) are fine. FileInfo.ReadSymbolicLink(string) is fine. new FileInfo/DirectoryInfo(string) does indeed need a second argument for whether to see through the symbolic link or not. Note that you should always use the OS facilities to do this when seeing through the link; that is call GetFileInformationByHandle on Windows.

You will have problems when it comes to Directory.GetFiles(), Directory.GetDirectories(), and Directory.GetFileSystemEntries(). Difficulties in making a reasonable API that didn't have abysmal performance lead to me abandoning the idea of making something that fit and actually designing a complete replacement for GetFilesystemEntries() that returned a structure with file name and metadata on it. I went as far as prepopulating the metadata structure with the information returned by the API call (readdir() returns a type hint).

Cases:

a symbolic link on *nix is clearly neither a file nor a directory, while on windows the link knows which it refers to.
some information is available from the directory walking API but it's not complete file information
a file entry could be a dangling symbolic link
some applications need to be able to use the API a way to avoid blowing up if the symbolic link refers to a host that is down; for this an other reasons a pathway through the API surface that doesn't resolve symbolic links must exist
Directory.Remove(symlinktree, true) is downright dangerous; I provided a replacement using proper native APIs so it actually works.

I didn't implement any way to get the final path of a symbolic link due to lack of a user. My library isn't prototype anymore; the entire API is in production use.

I'm not going to lie, my API design isn't perfect either; it could stand some improvement and potentially be a little less into itself, and also implement wildcard expansion (prior to relatively recent builds of Windows 10 you had to redo wildcard expansion in the client side anyway as the API call overmatched).

dotnet / runtime

Proposed API for symbolic links #24271

Rationale

Proposed API

Details

Updates