Support for libc++ std modules

[mstorsjo]

This is a draft for various fixes needed for using the libcxx C++23 import std modules (plus a bunch of other commits for testing it in the github actions pipelines, and for trying to speed up consecutive reruns).

Most of the changes here are to our packaging/setup overall, but there are a couple of patches needed for mingw-w64 - I'm trying to test and evaluate them before trying to upstream them.

See https://github.com/mstorsjo/llvm-mingw/actions/runs/8802495666 for artifacts with the latest round of patches (where the final artifacts might be done in a couple of hours), or https://github.com/mstorsjo/llvm-mingw/actions/runs/8799865568 for a completed set of artifacts from an earlier build (with an earlier iteration of the patches).

CC @huangqinjin @nolange.

[mstorsjo]

The necessary patches have been merged upstream, in llvm-project and mingw-w64 (and backported to the 18.x release branch of LLVM), so this should work now. If you're interested, have a look - I'll potentially merge these commits (except for the ones marked WIP, which are included only for testing) so they'd be included with the new llvm-mingw release for the LLVM 18.1.5 point release next week.

[cristianadam]

With modules in LLVM-MinGW we should be able to get more libraries ported https://arewemodulesyet.org/ 😀

[mstorsjo]

With modules in LLVM-MinGW we should be able to get more libraries ported https://arewemodulesyet.org/ 😀

:-)

Note that there are known issues with C++ module interfaces (not specifically to the C++23 std module) in e.g. https://github.com/mstorsjo/llvm-mingw/issues/421.

[huangqinjin]

The necessary patches have been merged upstream, in llvm-project and mingw-w64 (and backported to the 18.x release branch of LLVM), so this should work now.

@mstorsjo Great job! Could you please point out the corresponding upstream commits/PRs here for reference. Thanks!

[mstorsjo]

The necessary patches have been merged upstream, in llvm-project and mingw-w64 (and backported to the 18.x release branch of LLVM), so this should work now.

@mstorsjo Great job! Could you please point out the corresponding upstream commits/PRs here for reference. Thanks!

Thanks for having a look!

https://github.com/llvm/llvm-project/commit/b9b73814ad8acd55e88d0415f4110d272797697d was the only change for libcxx.

For mingw-w64, I did https://github.com/mingw-w64/mingw-w64/commit/30c0b623dbaf9dcc218c9068625f5e8dddd57aab, https://github.com/mingw-w64/mingw-w64/commit/847164bcf5a5a1454b444ce476a682f87cccce40 and https://github.com/mingw-w64/mingw-w64/commit/1652e9241b5d8a5a779c6582b1c3c4f4a7cc66e5.

[nolange]

I am a bit out of the loop, dont have the time to follow everything you 2 did (much appreciated btw.).

But I am wondering if clang-scan-deps is even used outside CMake at the moment? Because with CMake I would expect that you set CMAKE_\<LANG>_COMPILER_TARGET and CMake should figure everything out.

No need for wrappers with CMake, if you use the llvm-mingw toolchain with a fitting toolchain-file you do not need any symlinks or wrapper scripts (I am using it exclusively that way).

I am planing to to a PR one day, but this is what I actually use, needs to put into the files share/llvm-aarch64-w64-mingw32_toolchainfile.cmake share/llvm-armv7-w64-mingw32_toolchainfile.cmake share/llvm-i686-w64-mingw32_toolchainfile.cmake share/llvm-x86_64-w64-mingw32_toolchainfile.cmake:

if(CMAKE_VERSION VERSION_GREATER_EQUAL "3.20")
  set(_prefix "${CMAKE_CURRENT_LIST_DIR}/../")
  cmake_path(ABSOLUTE_PATH _prefix NORMALIZE)
  set(_filename "${CMAKE_CURRENT_LIST_FILE}")
  cmake_path(GET _filename FILENAME _filename)
else()
  get_filename_component(_prefix "${CMAKE_CURRENT_LIST_DIR}/../" ABSOLUTE)
  get_filename_component(_filename "${CMAKE_CURRENT_LIST_FILE}" NAME)
endif()

string(REGEX REPLACE "^llvm[_-]\(.*\)[_-]toolchain.*" "\\1" _arch "${_filename}")
string(REGEX REPLACE "^.*-mingw32\(.*\)" "\\1" _mingwtype "${_arch}")
string(REGEX REPLACE "-.*" "" _arch "${_arch}")
unset(_filename)

set(_exesuff)
if(CMAKE_HOST_WIN32)
set(_exesuff .exe)
endif()

set(CMAKE_SYSTEM_NAME Windows)
if(_arch)
  set(CMAKE_SYSTEM_PROCESSOR ${_arch})
else()
  set(CMAKE_SYSTEM_PROCESSOR ${CMAKE_HOST_SYSTEM_PROCESSOR})
endif()

set(CMAKE_ASM_COMPILER "${_prefix}bin/clang${_exesuff}")
set(CMAKE_C_COMPILER "${_prefix}bin/clang${_exesuff}")
set(CMAKE_CXX_COMPILER "${_prefix}bin/clang++${_exesuff}")
set(CMAKE_RC_COMPILER "${_prefix}bin/llvm-rc${_exesuff}")

set(CMAKE_CXX_FLAGS_INIT "-stdlib=libc++")

foreach(_lang ASM C CXX)
  set(CMAKE_${_lang}_COMPILER_TARGET ${CMAKE_SYSTEM_PROCESSOR}-w64-mingw32${_mingwtype})
  if(_mingwtype STREQUAL "uwp")
    set(CMAKE_${_lang}_FLAGS_INIT "${CMAKE_${_lang}_FLAGS_INIT} -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DWINAPI_FAMILY=WINAPI_FAMILY_APP -DUNICODE -D_UCRT")
  endif()
endforeach()
unset(_lang)

set(_linker lld)
# set(_linker_extra " -Wl,--undefined-version")
if(_mingwtype STREQUAL "uwp")
  set(_linker_extra "${_linker_extra} -Wl,-lwindowsapp -Wl,-lucrtapp")
endif()
set(CMAKE_EXE_LINKER_FLAGS_INIT "--start-no-unused-arguments -stdlib=libc++ -fuse-ld=${_linker} -rtlib=compiler-rt -unwindlib=libunwind${_linker_extra} --end-no-unused-arguments")
set(CMAKE_MODULE_LINKER_FLAGS_INIT "${CMAKE_EXE_LINKER_FLAGS_INIT}")
set(CMAKE_SHARED_LINKER_FLAGS_INIT "${CMAKE_EXE_LINKER_FLAGS_INIT}")
set(CMAKE_LINKER ${_prefix}bin/ld.${_linker})
unset(_linker_extra)
unset(_linker)

set(CMAKE_SYSROOT "${_prefix}${CMAKE_SYSTEM_PROCESSOR}-w64-mingw32")

unset(_mingwtype)
unset(_prefix)

set(ENV{PKG_CONFIG_SYSROOT_DIR} "${CMAKE_SYSROOT}")
if($ENV{PKG_CONFIG_LIBDIR})
    set(ENV{PKG_CONFIG_LIBDIR} "${CMAKE_SYSROOT}/usr/lib/pkgconfig:${CMAKE_SYSROOT}/usr/share/pkgconfig:$ENV{PKG_CONFIG_LIBDIR}")
else()
    set(ENV{PKG_CONFIG_LIBDIR} "${CMAKE_SYSROOT}/usr/lib/pkgconfig:${CMAKE_SYSROOT}/usr/share/pkgconfig")
endif()

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

[mstorsjo]

But I am wondering if clang-scan-deps is even used outside CMake at the moment? Because with CMake I would expect that you set CMAKE__COMPILER_TARGET and CMake should figure everything out.

Not sure how much other build systems even support C++ modules yet; once they do, they most probably will need to use clang-scan-deps in one form or another.

Yes, it’s certainly possible to avoid the need for wrappers, if invoking clang-scan-deps directly with the right argument - but then one also need to make sure to supply the -stdlib=libc++ option.

No need for wrappers with CMake, if you use the llvm-mingw toolchain with a fitting toolchain-file you do not need any symlinks or wrapper scripts (I am using it exclusively that way).

Sure, that’s one way of doing it. Whether it is the preferred way or not, probably mostly is personal preference. The main design behind llvm-mingw so far is the opposite; providing wrappers that contain all these settings, to avoid needing complicated cmake setup (yes, I know it’s mostly contained within the toolchain file - but still), where it is enough to supply the names of executables, or with some build systems, just supply the triple prefix.

As for making a PR to add such files, I’m a little undecided. It’s not my preferred way of using cmake. Adding it might not be harmful in itself even if I don’t use it, but it adds another file to maintain and test that it stays working. So I’m not quite sold on the concept.

[mstorsjo]

I’ll merge this PR now, so these additions are included in the new release that will be out soon.