Native Windows Support: MAX_PATH Limitation of PWSH

[sbixl]

If running bob under native Windows, bob (or rather the pwsh) fails due to MAX_PATH limitation. There is the possibility to change the settings in the windows registry (set LongPathsEnabled under Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem to 1) which is certainly possible but not recommended.

Attached you will find three repo cases. During analysis I have discovered, that switching back to cmd and calling cmd specific commands are working (see recipes). So I guess the limitation comes along with the .NET Core pwsh is based on.

Please download the attached three repo cases and extract the archive to c:/projects or a similar place.

1) Unpacking archives

a) Fails if you try to extract a archiv with a very deep folder structure (using bob internal unzipper). call: bob dev target/st::ext-stm32g4-rc1 b) Works if you try to extract a archiv with a very deep folder structure (using external tool bsdtar). call: bob dev target/st::ext-stm32g4-rc1 -DSWITCH_TO_CMD=True

2) Delete Directories

a) Fails if you try to delete a folder with a very deep folder structure from pwsh. call: bob dev target/st::ext-stm32g4-rc2 b) Works if you try to delete a folder with a very deep folder structure by switching from pwsh to cmd. call: bob dev target/st::ext-stm32g4-rc2 -DSWITCH_TO_CMD=True

3) Copying Directories

a) Fails if you try to recurse copy a folder with a very deep folder structure from pwsh. call: bob dev target/st::ext-stm32g4-rc3 b) Works if you try to recurse copy a folder with a very deep folder structure by switching from pwsh to cmd. call: bob dev target/st::ext-stm32g4-rc3 -DSWITCH_TO_CMD=True

[mahaase]

@sbixl could your problem be like this:

https://github.com/microsoft/msbuild/issues/53 https://github.com/microsoft/msbuild/issues/5331

the problem with the MAX_PATH issue is, that each application itself has to support this "new feature".

First you have to activate the feature in windows: Computer Configuration -> Admin Templates -> System -> FileSystem

Second, the application has to be rebuild with the activated feature, you can check this inside the manifest part of the application - I guess open the exe by text-editor should show you something like that:

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0" xmlns:asmv3="urn:schemas-microsoft-com:asm.v3" >
  <asmv3:application>
    <asmv3:windowsSettings xmlns="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
      <longPathAware>true</longPathAware>
    </asmv3:windowsSettings>
  </asmv3:application>
</assembly>

And finally, the application has to support the feature, really. Not like msbuild, there looks everything fine, also the manifest, but it doesn't work (check github links above). What I would say, that is magic, in the end you can do everything right, but it doesn't work. Finaly also windows 10 doesn't support this feature yet 👎.

Hope that helps, or I misunderstood the problem :)

BR.

[sbixl]

@mahaase Thank you for the hint but I basically mean something else. My description of the issue was maybe not pretty good enough. What I mean is, that the internals of bob can fail too, for example if you try to checkout and auto extract an archive with a very huge directory structure and long file names. Another example is when bob do a clean-up of the build or package directory where the internal "delete dir cmd" of bob can fail too.

I've attached two more repo cases (rc4 and rc5) where you can see much more in detail what happens.

4) call: bob dev target/st::ext-stm32g4-rc4 --> extracting the archive fails 5) call: bob dev target/st::ext-stm32g4-rc5 --> builds fine call: bob dev target/st::ext-stm32g4-rc5 --clean --> the cleanup of the build directory fails

I hope it's a little bit easier to understand now?

BR

[jkloetzke]

Thanks for the repo cases. I think the recommended solution is indeed to set the registry key. I couldn't repoduce issue rc4 until i realized that I already had the "LongPathsEnabled" registry key already set in the past. 😆

I will still have a look if we can do something about the bob-internal errors. But I don't think we can do anything the failures of pwsh or msbuild...

[jkloetzke]

I looked into it and its not pretty. :fearful: I'm sceptic that this is worth the effort to catch every path access in Bob and wrap it. Because that's what we would have to do. :see_no_evil: OTOH current Python versions work well when the long file name support is enabled in the registry. So I'm inclined to not fix that because it will certainly break again in the future and increase the maintenance burden considerably.

Objections?

[sbixl]

I agree, it's not worth the effort. But I think we should add a note in the documentation that it is recommended to set the registry key "LongPathsEnabled" to "1"?

[jkloetzke]

Yes, this sounds like a good idea. Still many thanks for providing the repo case.