Windows build fails to work with clang++

[SquareRoundCurly]

I have the weirdest issue. I have followed your clang-tutor and your llvm-tutor pages - great effort and very informative! Since in llvm-tutor there is a windows version for the project cmake file, I thought I could build your hello plugin on windows too. I managed to get something to build, but it only works with the -cc1 command, not with the -Xclang version. Furthermore, it only seems to work with clang.exe, couldn't get it to work with clang++.exe. I am probably doing something very wrong.

There are 2 things I have tried, when it comes to making the cmake file work: at first, I tried to make your cmake file work. It doesn't quite work out of the box. When cmake generates a visual studio solution, it links (correctly) the needed libs to the project, but also tries to find a "clang.lib", to no avail. When building LLVM/Clang (12), this library seems to stay in the build directory and not get copied over when cmake executes the install target. Fair enough, I just manually linked the odd one out clang.lib from the original build directory. Now the plugin compiles & works, but only with clang.exe and -cc1.

Second thing I tried, is that, on windows, this clang.lib is all that is needed for the linker, to build the plugin. All the libs in the llvm install dir/lib are unnecessary (so it would seem?). So I could build the whole hello plugin with just clang.lib and the include dirs at the install/include. However, the plugin still only works when called from clang.exe with -cc1.

Have you tried making hello plugin work on windows? Do you have any insight into why a plugin would run correctly with -cc1 and not with Xclang? Or with clang.exe but not with clang++.exe?

[SquareRoundCurly]

I also wrote up a question on stackoverflow regarding clang plugins on windows, further detailing my issue: link

[banach-space]

Hi @SquareRoundCurly !

Thank you for using my tutorials! I'm really glad that you've found them helpful.

Thank you for posting this question. Yes, there's a CMake file for Windows in llvm-tutor, but it's neither documented nor officially supported, because it's been quite problematic. I do have a Windows CI job set-up for it, but it's been failing for a while. Sadly it's very tricky for me to fix it as I have no access to a Windows machine. I'm hoping that one day somebody submits a fixing patch :)

Going back to your issue - I've noticed that you updated your SO question with some insights. You probably already know more about the issue than I do! IIUC, on Windows every library comes with a *.lib file that contains the list of symbols exported by the corresponding library. The linker will use that file to find the required symbol. So, if your plugin needs a symbol from Clang, that needs to be listed in one of the supplied *.lib files. However, there's an upper limit on the number of symbols that a library can export on Windows. Sadly, LLVM exports more than that (I imagine that it's similar in Clang). So, some symbols are available in the *.lib files, but not all.

• If you are lucky, the symbols that you require are available in one of the *.lib files.
• If you are unlucky, the symbols that you need are not listed in any of the *.lib files and there's not much you can do.

This is my vague understanding of the issue. Hopefully this clarifies things a bit.

You also mentioned some inconsistent behavior with clang++. That's very confusing. As you mentioned on SO, clang++ is just a symlink to clang. What you're describing is very very weird and unexpected. Sadly, without access to a Windows machine it's hard for me to dive deeper. On Linux things work as expected.

[SquareRoundCurly]

Thank you for the swift reply!

I also have posted in https://llvm.discourse.group, where I have found you again, linking to this repo. I feel it helped many people kick start their clang development.

If I recall correctly, windows supports 65535 symbols in a binary, which clang obviously overshoots. I think they even mentioned this limit as the reason behind the lacking windows support. However, when building clang on windows, a special library is also created, clang.lib. The reasoning behind it is that on windows, there is no mechanism for exporting symbols from an executable. So clang dumps the clang symbols in the special clang.lib. This is problematic for the reasons you have mentioned, the lack of symbol addresses, but also because cmake doesn't copy it over on INSTALL.

Getting unexpected errors due to this special lib not containing some symbols is something that I have not encountered yet but makes me extremely uncomfortable. I am sure that with some cmake magic, the massive amount of symbols clang would export could be divided into multiple clang.libs. Then this wouldn't be an issue. This issue really feels like the sword of Damocles; an unresolvable issue that can show up at any time and kill any project relying on clang.

As for the clang++ not working: I suspect that the default values for these cmake options are OFF (LLVM_INSTALL_BINUTILS_SYMLINKS, LLVM_INSTALL_CCTOOLS_SYMLINKS), so instead of creating symlinks, cmake just makes a copy of clang and renames it as clang++ as a post build step. Perhaps with symlinks, the plugins would work. Still, why clang++ refuses to use plugins as it's just a copy of clang is weird. I need further testing on the symlink option.

[banach-space]

Haha, thank you for advertising clang-tutor!

When I was trying to make things work on Windows, I found the discussion in this PR helpful: https://reviews.llvm.org/D18826 (and other things that followed). That refers to LLVM specifically, but I assume that that approach was extended to Clang. clang.lib is a separate thing that I've never looked into.

I have a couple of suggestion:

• Have you considered implementing clang-tidy checks instead? In practice, clang-tidy checks are just Clang plugins, but with much better support and infrastructure. For clang-tutor I wanted to keep things simple and hence focused on Clang.
• AFAIK, Clang developers use cfe-dev rather than Discourse. Trying posting there.
• You've been raising 2 issues in a single ticket. Perhaps it's worth separating them?

As for clang++, I suspect that Clang makes a distinction between clang and clang++ that's specific to Windows. Keep in mind that Clang will parse the name of the binary that was used and:

• assume C mode when using clang
• assume C++ mode when using clang++

It might be worth investigating that.

Also, you mentioned that you use LLVM/Clang 12, but the latest release is LLVM/Clang 11 :) I suspect that you meant ToT (top-of-trunk), but that's meaningless without mentioning the commit that you checked out. This maybe relevant (but doesn't have to). Btw, have you tried LLVM/Clang 11? That's what works for me on Linux. It's possible that something was broken since LLVM/Clang 11 was released.

[SquareRoundCurly]

I would much prefer to have clang plugins and clang tools would work as intended instead of hooking into clang-tidy. Thank you for providing the correct channel for communication, I'll be sure to check that out. Sorry for raising 2 issues in on ticket, working with open source projects is something new to me.

Why clang++ doesn't play nicely with plugins, I still have no clue.

The latest version of the documentation refers to itself as LLVM 12, however, I have reverted back to 11 as that is the last release as you have mentioned.

Some things to note: The windows limitation of 65,279 (2^16) symbols in a binary can be worked around. Using the /bigobj command line argument for msvc, 4,294,967,296 (2^32) symbols can be fit in to a binary, this is achieved by having multiple sections, each containing 2^16 number of symbols. LLVM has included this option in their cmake files, but only setting this option as needed. I am quite sure that clang can fit it's symbols in clang.lib (the library that is generated on windows since windows can not export symbols from executables). One thing to note is to use VC 2005+ toolchain for /bigobj. Why the option is only for specific files? I do not know, there has been a commit that would enable /bigobj globally on windows here, but I think that is only for the latest version, not for llvm 11.

Finally, a last thing I have found: when working with libTooling, the compile_commands.json doesn't generate with VS, however, using the ninja build system it does generate on windows. But ninja doesn't use /bigobj. So for a full build on windows, one would need to generate a ninja project, save the compile_commands.json, then generate a VS project and build with VS. That is a bit cumbersome.

I feel like many of the limitations of windows have been patched/worked around, I just don't understand why clang++ doesn't work nicely with plugins. Once I figure that out, I feel like clang and clang plugins might be viable on windows afterall.

[banach-space]

Sorry for raising 2 issues in on ticket, working with open source projects is something new to me.

No worries and don't mind at all! Sometimes it's very tricky to see that you're dealing with 2 separate issues. I made the suggestion specifically for cfe-dev. There's a lot of traffic there and sometimes it's tricky to draw people's attention. But it's really worth trying :)

I feel like many of the limitations of windows have been patched/worked around, I just don't understand why clang++ doesn't work nicely with plugins. Once I figure that out, I feel like clang and clang plugins might be viable on windows afterall.

Yes, sadly it's an area that could benefit from some love. Thank you for sharing your findings - it sounds like you're identifying a lot of things that could be improved. Have you considered contributing some patches to Clang?

[banach-space]

As Windows is currently not supported and there's been no traffic here, I'm closing this issue. Please open a new one if you have further questions.