cabal-testsuite failures on Windows due to the max path length

phadej commented 5 years ago

I trying disable few tests on windows & appveyor, they weren't run previously anyway.

[ ] cabal-testsuite\PackageTests\Regression\T5309\cabal.test.hs
[x] cabal-testsuite\PackageTests\Regression\T5677\cabal.test.hs
[ ] cabal-testsuite\PackageTests\Backpack\Includes3\cabal-internal.test.hs
[ ] cabal-testsuite\PackageTests\Backpack\Includes3\cabal-external.test.hs
[ ] cabal-testsuite\PackageTests\Backpack\Includes2\cabal-internal.test.hs
[ ] cabal-testsuite\PackageTests\Backpack\Includes2\cabal-external.test.hs
[ ] cabal-testsuite\PackageTests\Backpack\bkpcabal01\cabal.test.hs

phadej commented 5 years ago

@Mistuke

Does

C:\WORK\cabal-testsuite\PackageTests\Backpack\Includes3\cabal-internal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\Includes3-0.1.0.0\l\indef\Includes3-0.1.0.0-inplace-indef+AZNSCvlbjiwGW3rUT5hCYh\build\Includes3-0.1.0.0-inplace-indef+AZNSCvlbjiwGW3rUT5hCYh\autogen\: openBinaryTempFileWithDefaultPermissions:

Look like path length limitation? Is there anything one can do to make it not fail?

Mistuke commented 5 years ago

Yeah that path is way too long. Which ghc and Cabal is this?

Mistuke commented 5 years ago

You can try this before the build/testsuite https://github.com/raaz-crypto/raaz/commit/78a8459a65a04b72cfa45c0c6ccdff5a4cdfff6c

For newer cabal and ghc you can install ghc-jailbreak which would take care of it too.

phadej commented 5 years ago

@Mistuke, Cabal is master. In cabal-testsuite, we indeed have quite long CWD to begin with. Making X: is a nice trick, have to try that.

But the GHC used, is 8.6.5, will ghc-jailbreak work with it, or does one need 8.8?

phadej commented 5 years ago

I'd try ghc-jailbreak if it works, before making X: trick. I recall you blogged about it, but I cannot find that anymore, do you have a link?

Mistuke commented 5 years ago

It depends. I'm not quite sure who is issuing this error.. If it's coming from cabal or ghc. openBinaryTempFileWithDefaultPermissions sounds like it's coming from base openBinaryTempFileWithDefaultPermissions in which case only the X: trick will work until ghc 8.12 because the temp file function in base doesn't use the MAX_PATH workaround and ghc-jailbreak only patches the tools ghc depends on and not ghc itself.

Blog post is at https://hub.zhox.com/posts/bypassing-max_path-limit-on-windows/ but I think it won't work in this case.

Mistuke commented 5 years ago

Alternative is to backport the temp file creation from 8.12 to cabal As a compat fix.

phadej commented 5 years ago

If openBinaryTempFileWithDefaultPermissions can be backported without FFI, then I'm open to that idea.

It's used in one place; writeFileAtomic :: FilePath -> BS.ByteString -> NoCallStackIO (), so if that can be done somehow better on Win32, even better. (Now it's creating tmp file, and doing renameFile)

Mistuke commented 5 years ago

If openBinaryTempFileWithDefaultPermissions can be backported without FFI, then I'm open to that idea.

No it's mostly FFI. It gets complicated because openTemp and friends need to be atomic and work under threaded conditions. That means doing it in Haskell is tricky. Technically speaking at the moment cabal isn't threadsafe on Windows. Practically it probably is good enough because unlike hsc2hs and ghc you do a lot of work between invocations of openTemp.

So here are the options you have:

Shorten the paths using subst, no code change but not a "solution". As in, people will still hit it.
Backport openTemp and friends from GHC 8.10 and add some extra code to handle long paths, GCC won't like it so you'll need ghc-jailbreak as well untill GHC 8.10.
Backport openTemp and friends from GHC 8.12. Code is much simpler and handles long paths, but GCC won't like it until GHC 8.10 so you'll need ghc-jailbreak.
Change the temp path the function uses to something shorter.
XFAIL the test and re-enable it in two GHC releases or whenever the new I/O manager ships.

None of them very ideal solutions..

jneira commented 2 years ago

[ ] cabal-testsuite\PackageTests\Regression\T6906\cabal.test.hs also fails in windows for ghc-8.4.4 (see https://github.com/jneira/cabal/runs/5471916886?check_suite_focus=true#step:16:2319)

jneira commented 2 years ago

Trying to merge github workflows i noted lot of tests which works in ghc-8.6.5 with this invocation:

cabal v2-run cabal-testsuite:cabal-tests -- -j1 --with-cabal=dist-newstyle\build\x86_64-windows\ghc-${{ matrix.ghc }}\cabal-install-3.7.0.0\x\cabal\build\cabal\cabal.exe

does not work with this other generated by validate.sh:

D:\a\cabal\cabal\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-testsuite-3\build\cabal-tests\cabal-tests.exe --builddir=/d/a/cabal/cabal/dist-val-ghc-8.6.5/build/x86_64-windows/ghc-8.6.5/cabal-testsuite-3 --with-cabal=D:\a\cabal\cabal\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-install-3.7.0.0\x\cabal\build\cabal\cabal.exe -j2 --hide-successes

Enabling verbose in the succesful one i noted the test works with this log:

creating D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp

"C:\ProgramData\chocolatey\lib\ghc.8.6.5\tools\ghc-8.6.5\bin\ghc.exe" "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-odir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-hidir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-stubdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i" "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i." "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen" "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen" "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen" "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen" "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-optP-include" "-optPD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen\cabal_macros.h" "-hide-all-packages" "-Wmissing-home-modules" "-no-user-package-db" "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\home\.cabal\store\ghc-8.6.5\package.db" "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\packagedb\ghc-8.6.5" "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\package.conf.inplace" "-package-id" "base-4.12.0.0" "-package-id" "directory-1.3.3.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs" "-hide-all-packages"

[1 of 1] Compiling Main             ( MyCustomPreprocessor.hs, D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp\Main.o )

The broken one failed in the same step with:

creating D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp

"C:\ProgramData\chocolatey\lib\ghc.8.6.5\tools\ghc-8.6.5\bin\ghc-8.6.5.exe" "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-odir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-hidir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-stubdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i" "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i." "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen" "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen" "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen" "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen" "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-optP-include" "-optPD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen\cabal_macros.h" "-hide-all-packages" "-Wmissing-home-modules" "-no-user-package-db" "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\home\.cabal\store\ghc-8.6.5\package.db" "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\packagedb\ghc-8.6.5" "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\package.conf.inplace" "-package-id" "base-4.12.0.0" "-package-id" "directory-1.3.3.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs" "-hide-all-packages"

[1 of 1] Compiling Main             ( MyCustomPreprocessor.hs, D:\\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp\Main.o )

Assembler messages:

Fatal error: can't create D:\\\\a\\\\cabal\\\\cabal\\\\cabal-testsuite\\\\PackageTests\\\\CustomPreProcess\\\\cabal.dist\\\\work\\\\.\\\\dist\\\\build\\\\x86_64-windows\\\\ghc-8.6.5\\\\internal-preprocessor-test-0.1.0.0\\\\build\\\\my-custom-preprocessor\\\\my-custom-preprocessor-tmp\\Main.o: No such file or directory

`gcc.exe' failed in phase `Assembler'. (Exit code: 1)

The two ghc calls have exactly the same paths. The only difference is the bad one uses ghc-8.6.5.exe and the good one ghc.exe but i cant believe that could be the culprit.

Observe the double dash in the broken one maybe related?

jneira commented 2 years ago

Cant reproduce locally with

"D:\dev\ws\haskell\cabal\.\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-testsuite-3\build\cabal-tests\cabal-tests.exe"  --builddir="D:\dev\ws\haskell\cabal\.\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-testsuite-3" ".\cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs" --with-cabal="D:\dev\ws\haskell\cabal\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-install-3.7.0.0\x\cabal\build\cabal\cabal.exe"

:facepalm:

jneira commented 2 years ago

I am tempted to mark those tests as broken for ghc >= 8.6.5 as we are gonna test 8.10, 9.0 and hopefully 9.2 in windows but i would prefer not having to do. @Mistuke know you are busy but any kind of help would be much appreciated

Mistuke commented 2 years ago

The Paths aren't exactly the same:

The working one has

[1 of 1] Compiling Main             ( MyCustomPreprocessor.hs, D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp\Main.o )

and the broken one

[1 of 1] Compiling Main             ( MyCustomPreprocessor.hs, D:\\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp\Main.o )

Notice the difference D:\ vs D:\\. Cabal double escapes these. So the first path becomes:

> "D:\\a\\\\cabal\\\\cabal\\\\cabal-testsuite\\\\PackageTests\\\\CustomPreProcess\\\\cabal.dist\\\\work\\\\.\\\\dist\\\\build\\\\x86_64-windows\\\\ghc-8.6.5\\\\internal-preprocessor-test-0.1.0.0\\\\build\\\\my-custom-preprocessor\\\\my-custom-preprocessor-tmp\\Main.o".length
264

Which is just below the limit... and the broken

> "D:\\\\a\\\\cabal\\\\cabal\\\\cabal-testsuite\\\\PackageTests\\\\CustomPreProcess\\\\cabal.dist\\\\work\\\\.\\\\dist\\\\build\\\\x86_64-windows\\\\ghc-8.6.5\\\\internal-preprocessor-test-0.1.0.0\\\\build\\\\my-custom-preprocessor\\\\my-custom-preprocessor-tmp\\Main.o".length
266

which is above the limit. Something has done some weird double escaping here. don't quite understand the escaping logic.. It doesn't look consistent.

We only started patching the toolchain for long paths in GHC 8.10. With GHC 8.6 we patched GHC itself, but binutils is still bound.

For this particular test, if you can fix the over-escaping it would bring you back under.

D:/a/cabal/cabal/cabal-testsuite/PackageTests/CustomPreProcess/cabal.dist/work/./dist/build/x86_64-windows/ghc-8.6.5/internal-preprocessor-test-0.1.0.0/build/my-custom-preprocessor/my-custom-preprocessor-tmp/Main.o

is equivalent but much shorter at 214 chars.

jneira commented 2 years ago

many thanks @Mistuke, i had observed the double quoting behaviour, but i dont understand why invoking the cabal test suite with cabal run cabal-testsuite -- --test-args or invoking directly the executable under build dir with $(cabal-plan list-bin cabal-testsuite) --test-args could make that change

The full invocation of the working one is:

cabal v2-run cabal-testsuite:cabal-tests -- -j1 -v --with-cabal=dist-newstyle\build\x86_64-windows\ghc-8.6.5\cabal-install-3.7.0.0\x\cabal\build\cabal\cabal.exe

... without building the executable before that

The full invocation of the not working one is:

D:\a\cabal\cabal\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-testsuite-3\build\cabal-tests\cabal-tests.exe --builddir=D:\a\cabal\cabal\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-testsuite-3 --with-cabal=D:\a\cabal\cabal\dist-val-ghc-8.6.5\build\x86_64-windows\ghc-8.6.5\cabal-install-3.7.0.0\x\cabal\build\cabal\cabal.exe -j2 --hide-successes

... building the executable with cabal build in a msys2 bash console

So one hypothesis could be cabal run does some magic to avoid double escape.

I also tried to shorthen the dist-val-ghc-8.6.5 build dir to dist-val8.6.5 with no luck just in case.

Dont understand neither why cant reproduce it locally.

I tried the not working one in ci under msys2 bash, raw cmd and powershell and it did not work

jneira commented 2 years ago

which is above the limit. Something has done some weird double escaping here. don't quite understand the escaping logic.. It doesn't look consistent.

and the bad ghc call does not have any double slash in the ghc invocation, so it seems ghc is doing it depending on some stateful thing:

"C:\ProgramData\chocolatey\lib\ghc.8.6.5\tools\ghc-8.6.5\bin\ghc-8.6.5.exe"
  "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static" 
  "-outputdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" 
  "-odir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" 
  "-hidir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" 
  "-stubdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp" 
  "-i" "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-i." "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen"
  "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen"
  "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen"
  "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen"
  "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-optP-include"
  "-optPD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen\cabal_macros.h"
  "-hide-all-packages" "-Wmissing-home-modules" "-no-user-package-db"
  "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\home\.cabal\store\ghc-8.6.5\package.db"
  "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\packagedb\ghc-8.6.5"
  "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\package.conf.inplace"
  "-package-id" "base-4.12.0.0" "-package-id" "directory-1.3.3.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs" "-hide-all-packages"

good one, identical but for the ghc.exe instead ghc-8.6.5.exe, i also tried desperately use ghc.exein the bas one with no luck

"C:\ProgramData\chocolatey\lib\ghc.8.6.5\tools\ghc-8.6.5\bin\ghc.exe"
  "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static"
  "-outputdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-odir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-hidir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-stubdir" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-i" "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-i." "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen"
  "-iD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen"
  "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen"
  "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\global-autogen"
  "-ID:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\my-custom-preprocessor-tmp"
  "-optP-include"
  "-optPD:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\build\my-custom-preprocessor\autogen\cabal_macros.h"
  "-hide-all-packages" "-Wmissing-home-modules" "-no-user-package-db"
  "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\home\.cabal\store\ghc-8.6.5\package.db"
  "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\packagedb\ghc-8.6.5"
  "-package-db" "D:\a\cabal\cabal\cabal-testsuite\PackageTests\CustomPreProcess\cabal.dist\work\.\dist\build\x86_64-windows\ghc-8.6.5\internal-preprocessor-test-0.1.0.0\package.conf.inplace"
  "-package-id" "base-4.12.0.0" "-package-id" "directory-1.3.3.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs" "-hide-all-packages"

jneira commented 2 years ago

well maybe the output of the call is the same but the double slashes are removed printing out the log :thinking:

jneira commented 2 years ago

list of affected tests:

[ ] PackageTests\NewBuild\CmdTest\OptionsFlag\cabal.test.hs
[ ] PackageTests\NewBuild\CmdRun\ScriptWithProjectBlock\cabal.test.hs
[ ] PackageTests\NewBuild\CmdRun\ScriptRerun\cabal.test.hs
[ ] PackageTests\NewBuild\CmdRun\ScriptNoExtention\cabal.test.hs
[ ] PackageTests\NewBuild\CmdRun\ScriptLiterate\cabal.test.hs
[ ] PackageTests\NewBuild\CmdRun\Script\cabal.test.hs
[ ] PackageTests\NewBuild\CmdClean\Script\cabal.test.hs
[ ] PackageTests\NewBuild\CmdClean\Orphan\cabal.test.hs
[ ] PackageTests\NewBuild\CmdClean\Keep\cabal.test.hs
[ ] PackageTests\NewBuild\CmdBuild\ScriptRerun\cabal.test.hs
[ ] PackageTests\NewBuild\CmdBuild\ScriptBuildRun\cabal.test.hs
[ ] PackageTests\NewBuild\CmdBuild\ScriptBuildRepl\cabal.test.hs
[ ] PackageTests\NewBuild\CmdBuild\Script\cabal.test.hs
[ ] PackageTests\NewBuild\CmdBench\OptionsFlag\cabal.test.hs
[ ] PackageTests\FFI\ForeignOptsCxx\cabal.test.hs
[ ] PackageTests\FFI\ForeignOptsC\cabal.test.hs
[ ] PackageTests\CustomPreProcess\cabal.test.hs

Mistuke commented 2 years ago

and the bad ghc call does not have any double slash in the ghc invocation, so it seems ghc is doing it depending on some stateful thing:

Ah yes, prior to GHC 9, the versioned files used to be wrappers, and from 9, it's the unversioned one.

those wrappers are ancient and re-escape things. See https://github.com/ghc/ghc/blob/ghc-8.6.5-release/driver/utils/cwrapper.c That's the second level of escaping

looks like they don't escape /. That code needs to be smarted, I rewrote it at some point but never submitted it upstream.. You can try normalizing the paths to / instead of \ when calling GHC.

jneira commented 2 years ago

Ah yes, prior to GHC 9, the versioned files used to be wrappers, and from 9, it's the unversioned one.

those wrappers are ancient and re-escape things. See https://github.com/ghc/ghc/blob/ghc-8.6.5-release/driver/utils/cwrapper.c That's the second level of escaping

wow, million thanks for that, I would swear I tried it with the main ghc.exe but trying it again, that should fix the calls too, no?

Mistuke commented 2 years ago

wow, million thanks for that, I would swear I tried it with the main ghc.exe but trying it again, that should fix the calls too, no?

It should, for 9.x GHC HQ flipped which one is the main though. though 9.x shouldn't have a problem with long paths.

haskell / cabal

cabal-testsuite failures on Windows due to the max path length #6271