Support for long paths on Windows

[daniel-v]

I ran into an issue where I could not create/read FileSystemEntity on Win, while it worked just fine on other systems. I received 206 Errors. I tracked the issue down to the path being too long, it was more than the default 260 chars.

System Windows 10 x64

In the Windows API (with some exceptions discussed in the following paragraphs), 
the maximum length for a path is MAX_PATH, which is defined as 260 characters. 

Source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

By prefexing the path with \\?\ this specific issue was resolved in my case. Recommendation to do so came from here:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx

I see 2 ways to make sure that others won't need to dig into this:

1. Update the documentation for FileSystemEntity and let us know about this limitation on Windows
2. Update core SDK by prepending \\?\ for long path support (naively: whenever there is an Utf8ToWideScope.wide() invocation for paths, opt-in for long path support?)

The latter I'd prefer so that the Dart code would not need to include platform specific code.

[whesse]

There are disadvantages to automatically prepending \?\ to paths. I think it disables support for symbolic links and/or ".." to move up a directory, so this would be unexpected. I think new versions of windows are removing the path length restriction on normal paths, so this may be the better fix for the issue.

[daniel-v]

@whesse I am using 10 and the 260 path length restriction still applies on this machine (was a clean install, not an upgrade).

Starting in Windows 10, version 1607, MAX_PATH limitations have been removed from common Win32 file and directory functions. However, you must opt-in to the new behavior.

Opting in happens through regedit or The registry key can also be controlled via Group Policy at Computer Configuration > Administrative Templates > System > Filesystem > Enable NTFS long paths.

In either case, I have to do something about it manually.

I'll make a quick test case if prepending the prefix would break symlinks and '..' if it helps you.

[whesse]

It says in the documentation that it does disable .. and . and also the use of '/' instead of '\'. Especially the last is a problem, and it also says that it only applies to absolute paths, not relative paths. So it seems clear we cannot add //?/ by default, because it would cause unexpected behavior for users.

MSDN says: "These prefixes are not used as part of the path itself. They indicate that the path should be passed to the system with minimal modification, which means that you cannot use forward slashes to represent path separators, or a period to represent the current directory, or double dots to represent the parent directory. Because you cannot use the "\?\" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters."

[whesse]

It would be good to know if the Dart SDK and dart::io File fail to handle long paths in the case where NTFS long paths are enabled. There is a registry setting in Windows 10 that removes the 260 character limit, without requiring the //?/ prefix. If there is code in the SDK that limits file paths to 260 characters even when the OS is no longer doing it, we should consider fixing that.

Please try enabling long paths on Windows, and see if the failure continues, or changes: Here is how to enable long paths on Windows 10: How to enable paths longer than 260 characters in Windows 10 Hit the Windows key, type gpedit.msc and press Enter. Navigate to Local Computer Policy > Computer Configuration > Administrative Templates > System > Filesystem > NTFS. Double click the Enable NTFS long paths option and enable it. Another link says this location has changed: The value has moved from NTFS directly into Local Computer Policy > Computer Configuration > Administrative Templates > System > Filesystem https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing

[Tokenyet]

It would be good to know if the Dart SDK and dart::io File fail to handle long paths in the case where NTFS long paths are enabled. There is a registry setting in Windows 10 that removes the 260 character limit, without requiring the //?/ prefix. If there is code in the SDK that limits file paths to 260 characters even when the OS is no longer doing it, we should consider fixing that.

Please try enabling long paths on Windows, and see if the failure continues, or changes: Here is how to enable long paths on Windows 10: How to enable paths longer than 260 characters in Windows 10 Hit the Windows key, type gpedit.msc and press Enter. Navigate to Local Computer Policy > Computer Configuration > Administrative Templates > System > Filesystem > NTFS. Double click the Enable NTFS long paths option and enable it. Another link says this location has changed: The value has moved from NTFS directly into Local Computer Policy > Computer Configuration > Administrative Templates > System > Filesystem https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing

I'm using windows 11 and there is no such option, but I check the method 1 from the site.

Method 1: Edit The Registry By-Hand If You Are Comfortable Making Registry Changes Become an admin user on your system. Run regedit The key to change is located at: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled Set the key’s DWORD value to “1” Reboot to activate the change.

Its default value is 1, but I still need to add \\?\ to pass the test. It's wierd for new user on this kind of code:

if(await Directory(theLongPath).exist()) {
  await Directory(theLongPath).delete();
}

This will log something like, system can't find the specified path (系統找不到指定的路徑 on logs). It pass exist(), but delete() can't do the job. Take me whole day to find out It's the long path issue 😢

PS Z:\Project\McDedicatedServer\minecraft_cube_desktop\minecraft_cube_desktop\packages\forge_installer_repository> dart test
00:00 +0 -1: test\forge_installer_repository_ig_test.dart: ForgeInstallerRepository call process.start with correct arguments [E]
  FileSystemException: Deletion failed, path = 'Z:\Project\McDedicatedServer\minecraft_cube_desktop\minecraft_cube_desktop\packages\forge_installer_repository\test_temp\forge_installer_repository_test' (OS Error: 系統找不到指定的路徑。
  , errno = 3)
  test\forge_installer_repository_ig_test.dart 18:9  main.<fn>.<fn>

Hope dart will fix this for windows someday 🙏

Here is the workaround for the dart users.

import 'dart:io';
import 'package:path/path.dart' as p;

Future<void> deleteDirectory(String path) async {
  final isAbsolute = p.isAbsolute(path);
  final prefix = Platform.isWindows && isAbsolute? r'\\?\' : '';
  await Directory(prefix + path).delete(recursive: true);
}

[ThexXTURBOXx]

This seems to be a critical issue since there is not even a correct error message that helps users debug the problem. As can be seen above, this issue also affects pana when there are folders that are nested too deep.

After looking at harmonoid, I noticed that its creator has published the following package: https://pub.dev/packages/safe_local_storage

It is able to correctly handle long paths on Windows without any problems. Moreover, the property Local Computer Policy > Computer Configuration > Administrative Templates > System > Filesystem > NTFS indeed does not exist in Windows 11. However, long paths are also enabled on my computer using the registry entry mentioned by @Tokenyet.

This is something that should be considered a critical issue in my opinion. If there is no "good" solution to the issue (as explained above by @whesse), then at least there should be an appropriate error message such that users can handle this issue themselves.

[ThexXTURBOXx]

Another thing I just found: https://learn.microsoft.com/de-de/windows/win32/fileio/maximum-file-path-limitation Near the bottom, they say that specifying the following tag in the application manifest may solve the issue as well:

<application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>

Maybe this helps, however, I am not sure where to find the Dart application manifest or how to test it properly.

[jsroest]

fix: FileSystemException: Directory listing failed (activate brick with git on Windows) #496