FileStream.Lock() differs in behaviour between Windows and Linux using .NET 6.0

[ConcreteHatstand]

Describe the bug

Using a 'FileStream' object, I can open a file and can lock bytes within the file with the Lock() method. This will prevent locks (cause an exception) on another process attempting the same lock on the same file. This is fine and correct. If the first program (still holding its initial lock) attempts to set that lock again it will succeed on Linux but throw an exception on Windows. N.B. On Linux a second lock by a second program on the same file will succeed if the file is opened read-only. This is shared-lock behaviour probably from the fcntl() system call.

To Reproduce

This code requires a file 'testfile' to be present for the read-only case. It asks for input (anything) before the second lock to give the engineer time to kick off a second instance (a second process) of the program if required. $ cat Program.cs using System; using System.IO; using System.Text;

class Test {

public static void Main()
{
    string path = "testfile";

    //Open the stream for read or maybe write...
    using (FileStream fs = File.OpenRead(path))
    //using (FileStream fs = File.OpenWrite(path))
    {
        Console.WriteLine("Now doing the first lock...");
        fs.Lock((long) 3, (long) 1);
        Console.WriteLine("Repeating that lock once you type something...");
        string typed = Console.ReadLine();
        fs.Lock((long) 3, (long) 1);
        Console.WriteLine("Done the second lock..."); // Won't be seen on Windows cos the Lock explodes.
    }
}

}

Exceptions (if any)

• Windows (but not Linux) does this for both read-only and writable files. Unhandled exception. System.IO.IOException: The process cannot access the file because another process has locked a port ion of the file. : 'C:\Users\nixxxxa\XFHCF_demo_RW\testfile' at System.IO.Strategies.FileStreamHelpers.Lock(SafeFileHandle handle, Boolean canWrite, Int64 position, Int64 length)

  at System.IO.Strategies.OSFileStreamStrategy.Lock(Int64 position, Int64 length) at System.IO.Strategies.BufferedFileStreamStrategy.Lock(Int64 position, Int64 length) at System.IO.FileStream.Lock(Int64 position, Int64 length) at Test.Main() in C:\Users\nixxxxa\XFHCF_demo_RW\Program.cs:line 19

Further technical details

• Include the output of dotnet --info .NET SDK (reflecting any global.json): Version: 6.0.101 Commit: ef49f6213a

Runtime Environment: OS Name: Windows OS Version: 10.0.19042 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.101\

Host (useful for support): Version: 6.0.1 Commit: 3a25a7f1cc

.NET SDKs installed: 2.1.526 [C:\Program Files\dotnet\sdk] 3.1.100 [C:\Program Files\dotnet\sdk] 3.1.101 [C:\Program Files\dotnet\sdk] 5.0.404 [C:\Program Files\dotnet\sdk] 6.0.101 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.All 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs: https://aka.ms/dotnet-download

• Linux now... .NET SDK (reflecting any global.json): Version: 6.0.101 Commit: ef49f6213a

Runtime Environment: OS Name: rhel OS Version: 7 OS Platform: Linux RID: rhel.7-x64 Base Path: /home/rbbe/nxxx/dotnet-sdk-6.0.101/sdk/6.0.101/

Host (useful for support): Version: 6.0.1 Commit: 3a25a7f1cc

.NET SDKs installed: 6.0.101 [/home/rbbe/nxxx/dotnet-sdk-6.0.101/sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.1 [/home/rbbe/nxxx/dotnet-sdk-6.0.101/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.1 [/home/rbbe/nxxx/dotnet-sdk-6.0.101/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs: https://aka.ms/dotnet-download

• The IDE (VS / VS Code/ VS4Mac) you're running on, and its version Console apps, no IDE.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Describe the bug Using a 'FileStream' object, I can open a file and can lock bytes within the file with the Lock() method. This will prevent locks (cause an exception) on another process attempting the same lock on the same file. This is fine and correct. If the first program (still holding its initial lock) attempts to set that lock again it will succeed on Linux but throw an exception on Windows. N.B. On Linux a second lock by a second program on the same file will succeed if the file is opened read-only. This is shared-lock behaviour probably from the fcntl() system call. ### To Reproduce This code requires a file 'testfile' to be present for the read-only case. It asks for input (anything) before the second lock to give the engineer time to kick off a second instance (a second process) of the program if required. $ cat Program.cs using System; using System.IO; using System.Text; class Test { public static void Main() { string path = "testfile"; //Open the stream for read or maybe write... using (FileStream fs = File.OpenRead(path)) //using (FileStream fs = File.OpenWrite(path)) { Console.WriteLine("Now doing the first lock..."); fs.Lock((long) 3, (long) 1); Console.WriteLine("Repeating that lock once you type something..."); string typed = Console.ReadLine(); fs.Lock((long) 3, (long) 1); Console.WriteLine("Done the second lock..."); // Won't be seen on Windows cos the Lock explodes. } } } ### Exceptions (if any) - Windows (but not Linux) does this for both read-only and writable files. Unhandled exception. System.IO.IOException: The process cannot access the file because another process has locked a port ion of the file. : 'C:\Users\nixxxxa\XFHCF_demo_RW\testfile' at System.IO.Strategies.FileStreamHelpers.Lock(SafeFileHandle handle, Boolean canWrite, Int64 position, Int64 length) at System.IO.Strategies.OSFileStreamStrategy.Lock(Int64 position, Int64 length) at System.IO.Strategies.BufferedFileStreamStrategy.Lock(Int64 position, Int64 length) at System.IO.FileStream.Lock(Int64 position, Int64 length) at Test.Main() in C:\Users\nixxxxa\XFHCF_demo_RW\Program.cs:line 19 ### Further technical details - Include the output of `dotnet --info` .NET SDK (reflecting any global.json): Version: 6.0.101 Commit: ef49f6213a Runtime Environment: OS Name: Windows OS Version: 10.0.19042 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.101\ Host (useful for support): Version: 6.0.1 Commit: 3a25a7f1cc .NET SDKs installed: 2.1.526 [C:\Program Files\dotnet\sdk] 3.1.100 [C:\Program Files\dotnet\sdk] 3.1.101 [C:\Program Files\dotnet\sdk] 5.0.404 [C:\Program Files\dotnet\sdk] 6.0.101 [C:\Program Files\dotnet\sdk] .NET runtimes installed: Microsoft.AspNetCore.All 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] To install additional .NET runtimes or SDKs: https://aka.ms/dotnet-download - Linux now... .NET SDK (reflecting any global.json): Version: 6.0.101 Commit: ef49f6213a Runtime Environment: OS Name: rhel OS Version: 7 OS Platform: Linux RID: rhel.7-x64 Base Path: /home/rbbe/nxxx/dotnet-sdk-6.0.101/sdk/6.0.101/ Host (useful for support): Version: 6.0.1 Commit: 3a25a7f1cc .NET SDKs installed: 6.0.101 [/home/rbbe/nxxx/dotnet-sdk-6.0.101/sdk] .NET runtimes installed: Microsoft.AspNetCore.App 6.0.1 [/home/rbbe/nxxx/dotnet-sdk-6.0.101/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.1 [/home/rbbe/nxxx/dotnet-sdk-6.0.101/shared/Microsoft.NETCore.App] To install additional .NET runtimes or SDKs: https://aka.ms/dotnet-download - The IDE (VS / VS Code/ VS4Mac) you're running on, and its version Console apps, no IDE.

Author:	ConcreteHatstand
Assignees:	-
Labels:	`area-System.IO`, `untriaged`
Milestone:	-

[adamsitnik]

Hi @ConcreteHatstand

Our Unix implementation typically tries to mimic Windows-specific behavior that .NET has derived from .NET Framework which was Windows-only.

I can see that on Windows we are just calling LockFile method:

https://github.com/dotnet/runtime/blob/b84a05779aeadf7522e532a970094e5e36fbf5f9/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Windows.cs#L105

And its doc says:

Locks the specified file for exclusive access by the calling process.

Which makes me wonder why we are using shared locks on Unix for FileStreams that are not writeable:

https://github.com/dotnet/runtime/blob/b84a05779aeadf7522e532a970094e5e36fbf5f9/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs#L66

Perhaps we should be always using exclusive locks?

https://github.com/dotnet/runtime/blob/c2ec86b1c552ac8a1749f9f98e012f707e325660/src/libraries/Common/src/Interop/Unix/System.Native/Interop.LockFileRegion.cs#L13

I am going to apply the up-for-grabs label for this issue. The person who is willing to work on it should change the current implementation to always use exclusive locks:

- CheckFileCall(Interop.Sys.LockFileRegion(handle, position, length, canWrite ? Interop.Sys.LockType.F_WRLCK : Interop.Sys.LockType.F_RDLCK), handle.Path);
+ CheckFileCall(Interop.Sys.LockFileRegion(handle, position, length, Interop.Sys.LockType.F_WRLCK), handle.Path);

and simply run all the locking tests on Linux and see if they are all passing. I am afraid that one of these tests might start failing and tell us why this method is implemented the way it is.

[danmoseley]

@stephentoub you may have context here

[stephentoub]

It was using F_WRLCK in .NET 5: https://github.com/dotnet/runtime/blob/4aadfea70082ae23e6c54a449268341e9429434e/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.Lock.Unix.cs#L13 It was changed in https://github.com/dotnet/runtime/pull/44185 for .NET 6 by @Jozkee to address https://github.com/dotnet/runtime/issues/29173 and documented in https://docs.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/6.0/filestream-file-locks-unix.

[ConcreteHatstand]

Please be aware of issue https://github.com/dotnet/runtime/issues/29173 where exclusive locks were used even on read-only files. This is an illegal operation on Linux, you get a 'Bad File Descriptor' error.

[adamsitnik]

I currently can't see any way to get all the scenarios working. @ConcreteHatstand may I ask why are you using FileStream.Lock and what are you trying to achieve? Perhaps I will be able to suggest some other solution.

[AlgernonDanglepratt]

@adamsitnik What we have is a .NET implementation of a language that implements both file locking and record locking by means of syntax, and many applications written in that language rely heavily on that locking working correctly. Programs written in our language and compiled to .NET need to run on both Windows and Linux, and behave the same in the two environments. We could code round the differences by having different versions of our language runtime according to the base target platform... but the overheads (development, runtime performance, deploy-time considerations) are significant, and in all other respects we've come across .NET does a fantastic job of abstracting away those differences in base platform.

To be clear, @ConcreteHatstand and I work together.

[adamsitnik]

@AlgernonDanglepratt that is impressive!

May I ask why do you lock certain region(s) of file(s) rather than entire file(s)?

[Clockwork-Muse]

@AlgernonDanglepratt that is impressive!

May I ask why do you lock certain region(s) of file(s) rather than entire file(s)?

First thing that comes to my mind is certain legacy languages/database systems, like RPG for the IBM iSeries. You explicitly operate on rows in these situations (sometimes in a specific table order), and you have to specify explicit per-row locking behavior on read/write. If you locked the entire file you'd prevent anybody else from reading/updating the file, which might be something like your main customer database. IBM has been trying to move people away from this for multiple decades, now....

[AlgernonDanglepratt]

@Clockwork-Muse has the essence of our situation exactly right. "Proper" databases are not in practice used in the real world in every implementation of every situation where it might seem "obvious" to use them.

[Clockwork-Muse]

Potentially fixing this aside, I'm wondering if you should take a page out of IBM's book, and back things with an SQL database and translate read/write to SQL statements. For one thing, if your current system allows for index files, you're having to write/maintain the database code yourself, whereas backing it with an SQL database would move that maintenance burden to somebody else. And also allow people to write SQL statements, too.

[AlgernonDanglepratt]

We do have technology that allows our customers to make that choice... but it's their choice rather than ours.

[jozkee]

@ConcreteHatstand I think what you are asking is (correct me if I'm wrong): Unix: Use shared locks only for files with read-only permission (therefore https://github.com/dotnet/runtime/issues/29173) and use exclusive locks for everything else. Windows: should stay the same.

I misunderstood https://github.com/dotnet/runtime/issues/29173 and changed Lock to use a shared lock for files opened with FileAccess.Read.

[jozkee]

Not sure if we are already querying the file permissions, if we don't it will be adding overhead to FileStream.Lock.

Another possibility as you stated in https://github.com/dotnet/runtime/issues/29173 is to expose an API that allows you to specify which kind of Unix lock you want. The problem is that such API would be Unix-specific (I don't think it could do something on Windows) so it may be harder to sell to API review board.

[AlgernonDanglepratt]

@ConcreteHatstand has now retired, and I have been on leave, so I apologise for the delay in replying. I believe you have characterized the request correctly – thank you. The target is to give the same behaviour from the point of view of the programmer’s model, irrespective of the execution platform (ideally irrespective of file system, but that might not be possible).