Spurious issues with accessing files on network mounts

[falhumai96]

I have tested a code of my own in the following options:

• X64 VC++ MSVC
• X64 MinGW GCC
• X64 Linux
• X64 MinGW Clang

and my project compiled successfully in all the above targets except the latter. I am getting a bunch of the following errors:

In file included from <some HPP file>.HPP:<some file line number>:
In file included from <some HPP file>.HPP:<some file line number>:
In file included from <some HPP file>.HPP:<some file line number>:
In file included from <some HPP file>.HPP:<some file line number>:
...
<some HPP file>.HPP:<some file line number>:<some column number>: error: #include nested too deeply.

My project is a CMake project. Sometimes running parallel builds many times results in a successful build. Also, I am using a Ninja generator, and I am building directly on Windows.

As this is a private code, I cannot share code snippets here, but I am happy to have a private chat with you in a different chat channel regarding this matter.

[Biswa96]

Could you provide any minimal sample code or project to reproduce the issue?

[falhumai96]

let me try to create a sample project that can repro the same problem, and ill link you to it

[falhumai96]

For the time being, I have added @Biswa96 and @mstorsjo to my private repo for investigation.

[falhumai96]

Try to rebuild with GCC/G++ MinGW as CC/CXX vs Clang/Clang++ (e.g. both either from MSYS2 or clang/clang++ from release of llvm-mingw ... I tried both). Also, I noticed that Clang does not respect _setmode(_fileno(stdout), _O_U16TEXT); to set the output to unicode (I can't print using std::wcout for e.g.).

[Biswa96]

I can not reproduce your original issue with llvm-mingw. I have used the llvm-mingw-20230130-ucrt-x86_64.zip file, extract it in a directory, add that to PATH environment variable. Then compiled it with cmake + ninja + llvm-mingw without any problem.

[falhumai96]

Hmm weird. By the way, I am running the build from SSH to a Windows VM. Could this be a problem?

[falhumai96]

Also, what are your versions of CMake and Ninja?

[falhumai96]

My CMake is 3.26.0-rc3 My Ninja is 1.11.1

[Biswa96]

I'm using it in msys2 mingw environment in bare metal Windows 10.

• cmake: https://packages.msys2.org/base/mingw-w64-cmake
• ninja: https://packages.msys2.org/base/mingw-w64-ninja

I'm not sure if the environment have any effect on preprocessing CPP files.

[falhumai96]

Can you also try to change some simple logic in any of the files and try to rebuild? For some reason the build errors seems to be intermittent. Sometimes the build goes through, sometimes I see aforementioned bug above.

[Biswa96]

Tried it, can't reproduce any issue. If you want clean build environment the Docker image may help. Docker Hub link is in README file.

[falhumai96]

@Biswa96 If possible, can you try to build on Win 11 as well? I am using Win 11 pro VM btw.

[Biswa96]

I have tried in bare metal Windows 11 installation as well. Same result, no issue or compiler error.

[falhumai96]

I have just tried a Windows 10 Pro installation and same problem happens. Here is what I did:

• I download CMake 3.25.2 from official site: https://cmake.org/download/.
• I downloaded Ninja from official github release site from: https://github.com/ninja-build/ninja/releases/tag/v1.11.1.
• I download LLVM-MinGW from: https://github.com/mstorsjo/llvm-mingw/releases/tag/20220906.
• I installed CMake and set the path for all users.
• I put ninja somewhere and set path to it.
• Same thing for LLVM-MinGW's top bin folder.
• I set CC = x86_64-w64-mingw32-clang and CXX = x86_64-w64-mingw32-clang++.
• I did cmake \<project full path dir> -DCMAKE_INSTALL_PREFIX=\<some custom install dir full path> -GNinja
• I did "ninja install".

and same problem I see in my Win 10 pure installation.

[falhumai96]

@Biswa96 what do you mean by bare metal windows btw here? like IoT windows version?

[falhumai96]

can you please describe all the steps you have taken perform a clean build (also including how did you setup ur environment)? maybe I am missing something here.

[Biswa96]

bare metal means like normal installation on actual drive, no VM, no hypervisor. I am using msys2 (https://www.msys2.org/) project. But discussing about that would be off-topic here. You can use CI or container image to build your project with llvm-mingw.

[mstorsjo]

I cannot reproduce any such build problem with the repo you provided.

To get a third party reproducible setup, I would recommend trying to reproduce the issue in e.g. GitHub Actions builds. If you manage to show a failure there, anybody else with a copy of the same repo should be able to reproduce the bug in the same way.

[falhumai96]

I am getting different results in VirtualBox VM (above results) and bare metal Windows 11. On bare metal, I am now getting the following output:

cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E rm -f lib\lib<Some Lib>.a && C:\PROGRA~1\LLVMMI~1\20230130\msvcrt\bin\X8FE23~1.EXE qc lib\lib<Some Lib>.a  lib/CMakeFiles/<Some Dir>.dir/__/common/LongPathSupport.rc.res lib/CMakeFiles/<Some Dir>.dir/Common.cpp.obj lib/CMakeFiles/<Some Dir>.dir/<Some File>.cpp.obj lib/CMakeFiles/<Some Dir>.dir/<Some File>.cpp.obj lib/CMakeFiles/<Some Dir>.dir/<Some File>.cpp.obj && C:\PROGRA~1\LLVMMI~1\20230130\msvcrt\bin\X8396E~1.EXE lib\lib<Some Lib>.a && cd ."
C:\PROGRA~1\LLVMMI~1\20230130\msvcrt\bin\llvm-X8FE23~1: Invalid argument
ninja: build stopped: subcommand failed.

This is how I set the environment before calling CMake build and install (testing on release 20220906 (I found the same issue with Clang-15 and Clang-16) with msvcrt) [testing on PowerShell Core]:

$env:Path = "C:\Program Files\LLVM MinGW\20220906\msvcrt\bin;$env:Path"; $env:CC = "x86_64-w64-mingw32uwp-clang.exe"; $env:CXX = "x86_64-w64-mingw32uwp-clang++.exe".

Then, I do cmake <Project full path> -DCMAKE_INSTALL_PREFIX=<Install full path> -GNinja and ninja install on the build directory.

It is weird why I am getting different outputs on different setups (same Windows setup, just different way of running it (bare metal/VM)).

[alvinhochun]

Don't put anything in a path containing space characters. It is often a source of all kinds of weird issues.

[falhumai96]

But, C:\Program Files is by default having spaces, which is where more most binaries reside (e.g. default install path for CMake). Not handling spaces in my opinion is a bug (a bug that I did not find earlier when playing around with the toolchain like 2-3 years ago).

[falhumai96]

Hmm, having ninja and the LLVM MinGW toolchain in a path no spaces has no errors.

[cristianadam]

What happens if you use the 8.3 name in your powershell script?

Instead of:

$env:Path = "C:\Program Files\LLVM MinGW\20220906\msvcrt\bin;$env:Path";

use

$env:Path = "C:\PROGRA~1\LLVMMI~1\20230130\msvcrt\bin;$env:Path";

CMake uses 8.3 old dos naming for Ninja files in order to overcome any space issues.

Edit: I've noticed that there are two dates here in the LLVM MinGW paths, make sure you have the same version for the 8.3 test.

[alvinhochun]

If I am to guess, it is possible that the llvm-mingw wrapper doesn't know how to handle 8.3 module names.

[mstorsjo]

With the tools placed in a path with spaces, I can indeed reproduce issues. And the issues are most probably caused by the wrapper executables indeed.

If I am to guess, it is possible that the llvm-mingw wrapper doesn't know how to handle 8.3 module names.

Yeah, that looks like it. Or more precisely - when the tool has been executed as C:\PROGRA~1\LLVMMI~1\20230130\msvcrt\bin\X8FE23~1.EXE, the tool tries to inspect X8FE23~1.EXE to see what executable to invoke, but can't figure it out - while it would have expected something like x86_64-w64-mingw32-ar.exe (which it parses and then invokes llvm-ar.exe).

When I try it out, I run into a case where it fails to link, since it executes C:\code\16rc1\PATHWI~1\bin\C__~1.EXE, and the wrapper doesn't detect that C__~1.EXE is meant to be another name for c++.exe, i.e. it should link in C++ mode.

I guess some step that doesn't look at argv[0] but instead resolves the name of the executable, would be necessary. (The LLVM based tools do this internally.)

The current approach of using argv[0] would be better, though, if we would be using symlinks. E.g. it would allow us to have both clang.exe and clang++.exe be symlinks to clang-target-wrapper.exe; by looking at argv[0], we should be able to figure out which name the user used for the symlink. If we resolve it to the actual name of the exe, we'd only see clang-target-wrapper.exe.

Or are there other ways to ask the filesystem to reverse-map the mangled file name in argv[0] into a full pathname?

[mstorsjo]

(Or as a separate sidetrack - is there any way to ask the CMake generator to not flatten paths to 8.3 paths, trusting that the tools themselves can handle arguments/paths with spaces correctly?)

[cristianadam]

One can disable the 8.3 file naming on NTFS as seen at https://learn.microsoft.com/en-US/troubleshoot/windows-server/performance/stop-error-code-0x00000019

fsutil behavior set disable8dot3 1

If CMake and Ninja doesn't work afterwards then it's a valid CMake bug 🙂

[cristianadam]

I think it would be easier to use something like GetLongPathName and / or GetShortPathName.

[mstorsjo]

The current approach of using argv[0] would be better, though, if we would be using symlinks. E.g. it would allow us to have both clang.exe and clang++.exe be symlinks to clang-target-wrapper.exe; by looking at argv[0], we should be able to figure out which name the user used for the symlink. If we resolve it to the actual name of the exe, we'd only see clang-target-wrapper.exe.

It seems like I misremembered about this; we do see the name of the symlink instead of the main executable from GetModuleFileName.

The usecase I thought about was to fake the behaviour of symlinks without having them; you could do the equivalent of execl(/*file*/ "actual-executable.exe", /*argv[0]*/ "<triple-prefix>-toolname.exe", /*other-args*/...), where you pass a fake executable name in argv[0] - that would essentially get the effect of having symlinks, without needing to have physical files corresponding to that file name (which would be nice on Windows, where symlinks still are a bit problematic.) I don't think anything relies on that though. (And if it does, it'd be simple to add support for it, in the form of dirname(GetModuleFileName())/basename(argv[0]).)

I think it would be easier to use something like GetLongPathName and / or GetShortPathName.

Thanks, that seems to do the trick - see https://github.com/mstorsjo/llvm-mingw/commit/829aad161faea2f357ef840c130837a34e90d53b. (I'm currently test running this a bit; I'll push it to the master branch once it looks good. Freel free to comment on it!) I see that LLVM already was doing pretty much exactly the same too: https://github.com/llvm/llvm-project/blob/llvmorg-16.0.0-rc3/llvm/lib/Support/Windows/Process.inc#L209-L219

[falhumai96]

@mstorsjo if possible, can you also try the current LLVM MinGW wrappers in Windows inside VirtualBox? I have seen different results there. Not sure if it also applies o other VMs.

[falhumai96]

Try with a path with spaces too.

[mati865]

It doesn't matter if it's VM inside Hyper-V, VirtualBox or Bare Metal. The only difference comes to the OS and how you setup it.

[falhumai96]

Yes, I understand as this is the intended case. But, I tried with clean setup (same way I setup the type 2 hypervisor (VirtualBox)/bare metal/type 1 hypervisor (KVM)) and I found the former to have a much different compiler error than the latter two.

[falhumai96]

I keep on seeing random errors like:

FUNC_EXPORT std::variant<SomeString, SomeError> SomeClass::PathJoin(
            ^
In file included from Y:/Project/include/Class/Common.h:1:
In file included from Y:/Project/include/Class/Common.h:5
In file included from Y:/Project/include/Class/Common.h:2:
In file included from Y:/Project/include/Class/Common.h:2:
In file included from Y:/Project/include/Class/Common.h:2:
In file included from Y:/Project/include/Class/Common.h:2:
In file included from Y:/Project/include/Class/Common.h:2:
...
Y:/Project/include/Class/Common.h:14:1: error: unknown type name 'FUNC_EXPORT'
FUNC_EXPORT std::variant<std::vector<SomeString>, SomeError> SomeClass::ListDirectory(
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.

The errors are random. I have only seen such errors in a clean setup on VirtualBox Windows 10 and 11 (it is less likely that I have a wrong setup twice, but who knows). I just successfully made a build. Then, I added a space (literally just a space somewhere in a header), and I started seeing such errors. Then, I removed the space somewhere and I don't see it anymore. Intermittent errors. I am hoping that either @mstorsjo or anyone interested in this issue can look into this bug (I will share the private code with interested persons). As mentioned earlier, the code compiled fine in GCC MinGW, MSVC Visual C/C++, and Linux GCC (I would assume it compiles fine on macOS but I haven't tested it there too). I have only seen such weird intermittent compiler error in this toolchain. I have also moved the toolchain out of a folder with spaces.

[mstorsjo]

The errors are random. I have only seen such errors in a clean setup on VirtualBox Windows 10 and 11 (it is less likely that I have a wrong setup twice, but who knows). I just successfully made a build. Then, I added a space (literally just a space somewhere in a header), and I started seeing such errors. Then, I removed the space somewhere and I don't see it anymore. Intermittent errors. I am hoping that either @mstorsjo or anyone interested in this issue can look into this bug (I will share the private code with interested persons). As mentioned earlier, the code compiled fine in GCC MinGW, MSVC Visual C/C++, and Linux GCC (I would assume it compiles fine on macOS but I haven't tested it there too). I have only seen such weird intermittent compiler error in this toolchain. I have also moved the toolchain out of a folder with spaces.

If this issue is unrelated to the path with spaces issue, it doesn't sound like anything I've ever encountered. And as the issues are that random, I don't believe it helps to try to reproduce it just by compiling your code, even if I'd try to set up a build of it. It sounds mostly like there's some environmental factor in your setup (e.g. like paths with spaces, but something else) which ends up consistently present in your setups.

So we'd need to get the most reliable way to actually reproduce the issue outside of your setup. The ideal would be something like sharing a VM image where the problem is reproducible, but that's rather problematic in many ways...

The best way would be if it'd be possible to reproduce the errors in a controlled programmatic environment like github actions - have you tried setting that up? A script that downloads and unpacks an llvm-mingw toolchain, sets up compilation of your code, and does ninja; ninja clean; ninja; ninja clean a bunch of times?

Do you only use the msvcrt variants of the toolchains? I haven't test run those builds much at all, to be honest - maybe there's something that only happens in them?

[falhumai96]

I have not done the GitHub actions yet. I'll give it a try. Also, I've tried both msvcrt and ucrt.

On Fri., Mar. 10, 2023, 15:46 Martin Storsjö, @.***> wrote:

The errors are random. I have only seen such errors in a clean setup on VirtualBox Windows 10 and 11 (it is less likely that I have a wrong setup twice, but who knows). I just successfully made a build. Then, I added a space (literally just a space somewhere in a header), and I started seeing such errors. Then, I removed the space somewhere and I don't see it anymore. Intermittent errors. I am hoping that either @mstorsjo https://github.com/mstorsjo or anyone interested in this issue can look into this bug (I will share the private code with interested persons). As mentioned earlier, the code compiled fine in GCC MinGW, MSVC Visual C/C++, and Linux GCC (I would assume it compiles fine on macOS but I haven't tested it there too). I have only seen such weird intermittent compiler error in this toolchain. I have also moved the toolchain out of a folder with spaces.

If this issue is unrelated to the path with spaces issue, it doesn't sound like anything I've ever encountered. And as the issues are that random, I don't believe it helps to try to reproduce it just by compiling your code, even if I'd try to set up a build of it. It sounds mostly like there's some environmental factor in your setup (e.g. like paths with spaces, but something else) which ends up consistently present in your setups.

So we'd need to get the most reliable way to actually reproduce the issue outside of your setup. The ideal would be something like sharing a VM image where the problem is reproducible, but that's rather problematic in many ways...

The best way would be if it'd be possible to reproduce the errors in a controlled programmatic environment like github actions - have you tried setting that up? A script that downloads and unpacks an llvm-mingw toolchain, sets up compilation of your code, and does ninja; ninja clean; ninja; ninja clean a bunch of times?

Do you only use the msvcrt variants of the toolchains? I haven't test run those builds much at all, to be honest - maybe there's something that only happens in them?

— Reply to this email directly, view it on GitHub https://github.com/mstorsjo/llvm-mingw/issues/327#issuecomment-1464658269, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7DVCCUHFRFVIP34O23AFDW3O4MHANCNFSM6AAAAAAVP74IZE . You are receiving this because you authored the thread.Message ID: @.***>

[mstorsjo]

Regarding the issue with path with spaces; you can grab test toolchains from https://github.com/mstorsjo/llvm-mingw/actions/runs/4386709843, where that issue should be fixed. But I doubt that would make any difference for the intermittent random issues.

[falhumai96]

I am trying the test toolchain, and now I get the following errors:

-- The C compiler identification is Clang 16.0.0
-- The CXX compiler identification is Clang 16.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/bin/x86_64-w64-mingw32-clang.exe
-- Check for working C compiler: Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/bin/x86_64-w64-mingw32-clang.exe - broken
CMake Error at C:/Program Files/CMake/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
  The C compiler

    "Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/bin/x86_64-w64-mingw32-clang.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: C:/Users/testuser/DirectoryScannerBuildMingw/CMakeFiles/CMakeScratch/TryCompile-8djpch

    Run Build Command(s):C:/PROGRA~1/ninja/ninja.exe cmTC_de56e && [1/2] Building C object CMakeFiles/cmTC_de56e.dir/testCCompiler.c.obj
    [2/2] Linking C executable cmTC_de56e.exe
    FAILED: cmTC_de56e.exe 
    cmd.exe /C "cd . && Y:\windows-x86_64-toolchain\llvm-mingw-x86_64\bin\x86_64-w64-mingw32-clang.exe   CMakeFiles/cmTC_de56e.dir/testCCompiler.c.obj -o cmTC_de56e.exe -Wl,--out-implib,libcmTC_de56e.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
    ld.lld: error: undefined symbol: __set_app_type
    >>> referenced by ../crt/crtexe.c:0
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(pre_c_init)

    ld.lld: error: undefined symbol: __p__fmode
    >>> referenced by ../crt/crtexe.c:119
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(pre_c_init)

    ld.lld: error: undefined symbol: __p__commode
    >>> referenced by ../crt/crtexe.c:120
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(pre_c_init)

    ld.lld: error: undefined symbol: _setargv
    >>> referenced by ../crt/crtexe.c:125
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(pre_c_init)

    ld.lld: error: undefined symbol: _matherr
    >>> referenced by ../crt/crtexe.c:129
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(pre_c_init)

    ld.lld: error: undefined symbol: __mingw_setusermatherr
    >>> referenced by ../crt/crtexe.c:129
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(pre_c_init)

    ld.lld: error: undefined symbol: __getmainargs
    >>> referenced by ../crt/crtexe.c:146
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(pre_cpp_init)

    ld.lld: error: undefined symbol: _gnu_exception_handler
    >>> referenced by ../crt/crtexe.c:257
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)
    >>> referenced by Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(.xdata)
    >>> referenced by Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(.xdata)

    ld.lld: error: undefined symbol: _amsg_exit
    >>> referenced by ../crt/crtexe.c:234
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: _initterm
    >>> referenced by ../crt/crtexe.c:239
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)
    >>> referenced by ../crt/crtexe.c:246
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: __guard_dispatch_icall_fptr
    >>> referenced by ../crt/crtexe.c:254
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: _pei386_runtime_relocator
    >>> referenced by ../crt/crtexe.c:256
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: _set_invalid_parameter_handler
    >>> referenced by ../crt/crtexe.c:261
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: _fpreset
    >>> referenced by ../crt/crtexe.c:263
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: malloc
    >>> referenced by ../crt/crtexe.c:327
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)
    >>> referenced by ../crt/crtexe.c:333
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: strlen
    >>> referenced by ../crt/crtexe.c:332
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: memcpy
    >>> referenced by ../crt/crtexe.c:334
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: __main
    >>> referenced by ../crt/crtexe.c:266
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)
    >>> referenced by CMakeFiles/cmTC_de56e.dir/testCCompiler.c.obj:(main)

    ld.lld: error: undefined symbol: _cexit
    >>> referenced by ../crt/crtexe.c:277
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: undefined symbol: exit
    >>> referenced by ../crt/crtexe.c:274
    >>>               Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o:(__tmainCRTStartup)

    ld.lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
    clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:2 (project)

[falhumai96]

I guess the CRT is not linked with it.

[longnguyen2004]

Is Y: a network drive? I have a feeling that it might affect the compiler somehow

[falhumai96]

Y:/ is indeed a network drive. This is a shared drive between my host and my VM.

On Sun., Mar. 12, 2023, 00:48 Long Nguyen, @.***> wrote:

Is Y: a network drive? I have a feeling that it might affect the compiler somehow

— Reply to this email directly, view it on GitHub https://github.com/mstorsjo/llvm-mingw/issues/327#issuecomment-1465128505, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7DVCAFWIMEMW5NF7QRDADW3WEXBANCNFSM6AAAAAAVP74IZE . You are receiving this because you authored the thread.Message ID: @.***>

[falhumai96]

Ok, I figured out the root cause of the intermittent problem. It happens because the root project I am compiling is in a network drive and mounted, in this example, in Y:. Also, the project is a C++ project (with some C headers). One more thing I noticed as well (I tried it with the released Clang 14 and 15 toolchains), is that the default C++ exceptions support is not enabled by default, nor which standard it will use for C++, so I need to define CXX to be, for instance, <prefix>clang++ -fexceptions -lc++ or <prefix>clang++ -fexceptions -lstdc++. I have only noticed this problem in this toolchain.

btw, kudos to @longnguyen2004 for hinting at the shared drive problem above :smiley:!

[falhumai96]

I will try other methods of mounting my host's share inside my VM for cross build and see if this is the problem with VirtualBox's directory sharing or any way to share the drive is problematic and the problem is within the toolchain.

[mstorsjo]

Ok, I figured out the root cause of the intermittent problem. It happens because the root project I am compiling is in a network drive and mounted, in this example, in Y:.

Ok, so Clang reading source files mounted via VirtualBox's directory sharing has got intermittent random issues? That sounds plausible and explains why it doesn't show up anywhere else.

Also, the project is a C++ project (with some C headers). One more thing I noticed as well (I tried it with the released Clang 14 and 15 toolchains), is that the default C++ exceptions support is not enabled by default, nor which standard it will use for C++, so I need to define CXX to be, for instance, <prefix>clang++ -fexceptions -lc++ or <prefix>clang++ -fexceptions -lstdc++. I have only noticed this problem in this toolchain.

If the toolchain is in a path with spaces, or something else that causes CMake to invoke the compiler via obfuscated 8.3 file names, this sounds understandable - the compiler wrapper didn't know it's being invoked as clang++. The patch I made above should fix that.

I'll take a look at why you're getting the cmake errors with the toolchain from github actions, if that's caused by the long filenames patch or what's at play there...

[mstorsjo]

I am trying the test toolchain, and now I get the following errors:

-- The C compiler identification is Clang 16.0.0
-- The CXX compiler identification is Clang 16.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/bin/x86_64-w64-mingw32-clang.exe
-- Check for working C compiler: Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/bin/x86_64-w64-mingw32-clang.exe - broken
CMake Error at C:/Program Files/CMake/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
  The C compiler

    "Y:/windows-x86_64-toolchain/llvm-mingw-x86_64/bin/x86_64-w64-mingw32-clang.exe"

  is not able to compile a simple test program.

I managed to reproduce this issue - but that's not caused by the recent patch to resolve 8.3 pathnames to long path names with GetLongPathName, but it's caused by the fact that you placed the toolchain itself in a network mount directory. I can reproduce that same issue with older toolchains too, if you have them placed on a network share. (In my case I tested it with a local folder mounted on a remote machine via a RDP client.)

With that issue so easily reproducible, I should hopefully be able to track down that issue. I believe it's a core issue in LLVM somewhere - unfortunately it's probably too late for 16.0.0 for such an bugfix right now though, even if I would have it resolved right now...

[falhumai96]

Awesome, thank you for tracking this issue.

I am also looking forward to have a fix to this issue as well https://github.com/mstorsjo/llvm-mingw/issues/222, as this is a critical problem in another project I am also working on and the project supports STD IO with Unicode characters. This bug behaviour was not observed in GCC MinGW toolchain nor MSVC Visual Studio build tools, only in the LLVM MinGW toolchain.

[falhumai96]

Also, with respect to long path support for Windows native wrappers, why not add the longPathAware manifest file to the C/C++ wrapper (and also leverage std::filesystem for file manipulation, so to minimize path bugs)? imo it is a cleaner solution to, e.g., \\?\ hacky solution.

[mstorsjo]

as well #222, as this is a critical problem in another project I am also working on and the project supports STD IO with Unicode characters. This bug behaviour was not observed in GCC MinGW toolchain nor MSVC Visual Studio build tools, only in the LLVM MinGW toolchain.

Yeah, I'm aware of that issue, but I don't really have any clue where to start digging on it, and whether it requires deep refactorings of libc++ streams internals to fix it or not.

Also, with respect to long path support for Windows native wrappers, why not add the longPathAware manifest file to the C/C++ wrapper? imo it is a cleaner solution to, e.g., \\?\ hacky solution.

FWIW, the fix GetLongPathName isn't about long (>260 char) path names, but about converting 8.3 garbled filenames into proper filenames - I'm not using the \\?\ solution anywhere there right now. (I'm not sure if the whole 260 char limit ever hits any use within the wrapper. But if it does, that's a totally different thing than the new call to GetLongPathName to fix 8.3 paths.) If necessary, the longPathAware manifest might be a reasonable way to fix it, yes.

(and also leverage std::filesystem for file manipulation, so to minimize path bugs)?

The wrapper is intentionally kept minimal - as the toolchain installations usually don't use symlinks, the wrapper binary gets duplicated many many times, so I want it to stay as small as possible.

[longnguyen2004]

btw, kudos to @longnguyen2004 for hinting at the shared drive problem above 😃!

You'll know that "network path = bad for LLVM" if you've seen #233 😁

[mstorsjo]

btw, kudos to @longnguyen2004 for hinting at the shared drive problem above 😃!

You'll know that "network path = bad for LLVM" if you've seen #233 😁

Indeed, kudos for pointing that out!

Anyway, I've got some progress on figuring it out. The problem seems to be that LLVM trusts what Windows promises to us, but some filesystems don't seem to fulfill this.

I tested a couple of network file systems; when mounting a Samba share (from a Linux server), there's no problem. When mounting a shared folder in a VM in VMware Fusion, there's no problem. When mounting a shared folder via Microsoft Remote Desktop, there is a problem though - and apparently in shared folders mounted via VirtualBox too.

When LLD reads files, it checks the unique ID of the files, in order to deduplicate the files. On the problematic filesystems, the unique IDs aren't unique (or stable for that matter). See https://learn.microsoft.com/en-us/windows/win32/api/fileapi/ns-fileapi-by_handle_file_information - "The identifier (low and high parts) and the volume serial number uniquely identify a file on a single computer".

In a test run with linking with the toolchain in a shared folder, I saw this:

ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crt2.o returned 16777220/18446708398794096656
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/crtbegin.o returned 16777220/18446708398730637328
ld.lld: getUniqueID for hello.o returned 1510595989/18577348462919738
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libmingw32.a returned 16777220/18446708398706917392
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/lib/clang/16/lib/windows/libclang_rt.builtins-x86_64.a returned 16777220/18446708398932705296
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libunwind.dll.a returned 16777220/18446708398780104720
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libmoldname.a returned 16777220/18446708398594809872
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libmingwex.a returned 16777220/18446708398645280784
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libmsvcrt.a returned 16777220/18446708398594809872
ld.lld: skipping enqueueing path for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libmsvcrt.a
ld.lld: getUniqueID for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libadvapi32.a returned 16777220/18446708398594809872
ld.lld: skipping enqueueing path for z:/llvm-mingw-x86_64/x86_64-w64-mingw32/lib/libadvapi32.a

Note how it returned the same unique ID for libmsvcrt.a and libadvapi32.a as it already had returned for libmoldname.a. So LLVM concludes that libmsvcrt.a is a file that has already been processed, and ignores it.

When rerunning the same command, without changing anything, the unique IDs for the files have changed to other numbers again...

So the implementations of these remote filesystems come up with random, not unique numbers for these files, and LLVM's implementation relies on these filesystem provided unique ID numbers to disambiguate files from each other.