`Cabal` is broken after updating `mingw-w64-x86_64-crt-git` package

[vsafonkin]

Hi, I can't build sample haskell application on Windows with MSYS2 by Cabal tool. It happened after update mingw-w64-x86_64-crt-git package.

We are maintainers of virtual-environments repository and this issue blocked the rollout of the new Windows images for GitHub Actions.

Sample repro: hello world application: https://github.com/vsafonkin/sample-msys-crt-git-package-bug Successful build with mingw-w64-x86_64-crt-git package version 9.0.0.6373.5be8fcd83-1: https://github.com/vsafonkin/sample-msys-crt-git-package-bug/actions/runs/1876488973 Failed build with mingw-w64-x86_64-crt-git package version 9.0.0.6373.5be8fcd83-2: https://github.com/vsafonkin/sample-msys-crt-git-package-bug/actions/runs/1876499498

Error:

Failed to build exe:haskell-say-exe from haskell-say-1.0.0.0.
Build log (
C:\cabal\logs\ghc-9.2.1\haskell-say-1.0.0.0-7116a2e9f87[5](https://github.com/vsafonkin/sample-msys-crt-git-package-bug/runs/5275068585?check_suite_focus=true#step:4:5)0afc93c33140953997584183[6](https://github.com/vsafonkin/sample-msys-crt-git-package-bug/runs/5275068585?check_suite_focus=true#step:4:6)ecf.log
):
Preprocessing executable 'haskell-say-exe' for haskell-say-1.0.0.0..
Building executable 'haskell-say-exe' for haskell-say-1.0.0.0..
[1 of 2] Compiling Main             ( app\Main.hs, dist\build\haskell-say-exe\haskell-say-exe-tmp\Main.o )
[2 of 2] Compiling Paths_haskell_say ( dist\build\haskell-say-exe\autogen\Paths_haskell_say.hs, dist\build\haskell-say-exe\haskell-say-exe-tmp\Paths_haskell_say.o )
cabal.exe: Failed to build exe:haskell-say-exe from haskell-say-1.0.0.0. See
Linking dist\\build\\haskell-say-exe\\haskell-say-exe.exe ...
the build log above for details.
C://tools//ghc-9.2.1//mingw//bin/ld.exe: C:/tools/ghc-9.2.1/mingw/bin/../lib/gcc/x[8](https://github.com/vsafonkin/sample-msys-crt-git-package-bug/runs/5275068585?check_suite_focus=true#step:4:8)6_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/crt2.o:crtexe.c:(.rdata$.refptr.mingw_app_type[.refptr.mingw_app_type]+0x0): undefined reference to `mingw_app_type'

C://tools//ghc-[9](https://github.com/vsafonkin/sample-msys-crt-git-package-bug/runs/5275068585?check_suite_focus=true#step:4:9).2.1//mingw//bin/ld.exe: C:/tools/ghc-9.2.1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/[10](https://github.com/vsafonkin/sample-msys-crt-git-package-bug/runs/5275068585?check_suite_focus=true#step:4:10).2.0/../../../../x86_64-w64-mingw[32](https://github.com/vsafonkin/sample-msys-crt-git-package-bug/runs/5275068585?check_suite_focus=true#step:4:32)/lib/../lib/crt2.o:crtexe.c:(.rdata$.refptr.mingw_initcharmax[.refptr.mingw_initcharmax]+0x0): undefined reference to `mingw_initcharmax'
C://tools//ghc-9.2.1//mingw//bin/ld.exe: C:/tools/ghc-9.2.1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/crt2.o:crtexe.c:(.rdata$.refptr.mingw_initltssuo_force[.refptr.mingw_initltssuo_force]+0x0): undefined reference to `mingw_initltssuo_force'
C://tools//ghc-9.2.1//mingw//bin/ld.exe: C:/tools/ghc-9.2.1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/crt2.o:crtexe.c:(.rdata$.refptr.mingw_initltsdyn_force[.refptr.mingw_initltsdyn_force]+0x0): undefined reference to `mingw_initltsdyn_force'
C://tools//ghc-9.2.1//mingw//bin/ld.exe: C:/tools/ghc-9.2.1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/crt2.o:crtexe.c:(.rdata$.refptr.mingw_initltsdrot_force[.refptr.mingw_initltsdrot_force]+0x0): undefined reference to `mingw_initltsdrot_force'
collect2.exe: error: ld returned 1 exit status
`gcc.exe' failed in phase `Linker'. (Exit code: 1)

[al-cheb]

cc @eine @lazka

[Biswa96]

Have you tried rebuilding the Cabal tool (or the ghc) with new crt package?

[vsafonkin]

Have you tried rebuilding the Cabal tool with new crt package?

Hi @Biswa96, we install already built Cabal and ghc by Chocolatey package manager.

[al-cheb]

cc @hasufell

[al-cheb]

@Biswa96 , We use the following link to install haskell - 'https://downloads.haskell.org/~ghc/9.2.1/ghc-9.2.1-x86_64-unknown-mingw32.tar.xz'

[hasufell]

I can reproduce this. The way you can go around this issue is to disable msys2 integration in cabal:

1. remove extra-include-dirs: ... from the global cabal config (I think on virtual-machines it's in C:\cabal\config? Otherwise it's somewhere in AppData)
2. remove extra-lib-dirs: ... as well

For user this means that cabal won't be able to see msys2 installed packages and can't link to them.

I was under the impression that these settings are not enabled anyway: https://github.com/actions/virtual-environments/blob/60dc1b5f7a1a046eceaea0e5667beefdead26d65/images/win/scripts/Installers/Install-Haskell.ps1#L41

Invoke-Command -ScriptBlock ([ScriptBlock]::Create($bootstrapHaskell)) -ArgumentList $false, $true, $true, $false, $true, $false, $false, $ghcupPrefix, "", $msysPath, $cabalDir

The 5th argument NoAdjustCabalConfig should instruct the installer to not perform this config change.

It's possible that that is a bug in the installer: https://gitlab.haskell.org/haskell/ghcup-hs/-/blob/ab36d4418e508fdd9d5955c7ee7c5996a7c4d759/scripts/bootstrap/bootstrap-haskell#L528 (this should probably return 0).

However, this will still leave users CI broken that need msys2 libdir/include dirs.

[naveen521kk]

From the above log posted it looks like a external mingw version (I think included in ghc) is used. I'm just wondering how this build failure related to the update in mingw-w64-x86_64-crt-git when tools provided from MSYS2 isn't used. Can you try whether not installing mingw-w64-x86_64-crt-git fixes something?

Edit: Just read the previous comment and it makes sense now

[hasufell]

From the above log posted it looks like a external mingw version (I think included in ghc) is used.

Yes, GHC ships a very small toolchain with it, not a full msys2 distribution. And it's hardcoded to use that one afaik.

[hasufell]

@Mistuke any idea why we get this linker error only with the new package?

[Biswa96]

If I do not update crt in CI everything is OK. Can anyone test this after rebuilding ghc with newer crt, headers and gcc? Things changed a lot.

BTW, those symbols come from here...

$ grep -i mingw_initltssuo_force /ucrt64/lib/libmingw32.a
Binary file /ucrt64/lib/libmingw32.a matches

[naveen521kk]

any idea why we get this linker error only with the new package?

Most likely related to https://github.com/msys2/MINGW-packages/pull/10634

[vsafonkin]

If I do not update crt in CI everything is OK. Can anyone test this after rebuilding ghc with newer crt, headers and gcc? Things changed a lot.

@Biswa96, the new version of Windows runner has updated toolchain package and crt 9.0.0.6373.5be8fcd83-2 and it doesn't work with ghc 9.2.1 and the latest cabal. So we can't rollout this runner version at the moment.

[hasufell]

I believe with this https://gitlab.haskell.org/haskell/ghcup-hs/-/commit/2c583bcae9e333693b9bd8fc353bb0dcb2af71df (and an updated virtual image, which now would NOT adjust extra-include-dirs and extra-lib-dirs by default)... it should at least work for simple projects. But any user who needs msys2 libraries will still hit this.

[Mistuke]

@Mistuke any idea why we get this linker error only with the new package?

Because the namespace of those internal symbols were changed upstream, see https://www.mail-archive.com/mingw-w64-public@lists.sourceforge.net/msg18885.html

But this only works when every package that depends on this is updated as well. Can you rerun the command with -optl-Wl,-Map=a.map and put the resulting a.map file in a gist. I need to understand where it's pulling symbols from. In principle you shouldn't be able to override the crt location.

[hasufell]

@Mistuke https://gist.githubusercontent.com/hasufell/af76f345fad10f877a7ad5d0771d1222/raw/4dcf885e2c917ac0cdd25137c93037bd08474229/gistfile1.txt

[Mistuke]

@hasufell thanks. This ABI incompatibility isn't easy to fix post mortem, but can be easily worked around.

The error is coming from

 .bss           0x0000000000532370       0x10 C:\ghcup\msys64\mingw64\lib/libmingw32.a(lib64_libmingw32_a-tlssup.o)
                0x0000000000532370                __mingw_initltssuo_force
                0x0000000000532378                __mingw_initltsdrot_force
                0x000000000053237c                _tls_index
                0x0000000000532374                __mingw_initltsdyn_force

Though the symbol is internal it seems it's use is spread across more than one package. The problem is that the folder libmingw32.a is in isn't explicitly overridden by the GHC driver. This means that paths cabal pass it can override it as it's an implicit path.

We should add the libraries path to the driver, can you file a ticket on the GHC bug tracker.

There's only one workaround I can think of that'll work for both ghc and ghci which is to add this folder as well to the extra lib-dirs for cabal. You must add this before anything else

This folder is essentially $ghc_top_install_dir/mingw/x86_64-w64-mingw32/lib/ that should fix it and prevent future ABI breaks like these.

[hasufell]

https://gitlab.haskell.org/ghc/ghc/-/issues/21111

[hasufell]

There's only one workaround I can think of that'll work for both ghc and ghci which is to add this folder as well to the extra lib-dirs for cabal. You must add this before anything else

I see a problem with this... because it depends on the GHC version in use. global extra-lib-dirs isn't smart enough for this.