Restriction on std modules looks not reasonable

ChuanqiXu9 commented 1 year ago

Xmake Version

xmake v2.7.6+master.7f8eefb8a

Operating System Version and Architecture

linux

Describe Bug

I am trying to compile https://github.com/alibaba/async_simple/tree/CXX20Modules with xmake.

// xmake.lua
add_rules("mode.debug", "mode.release")

target("std")
    set_kind("static")
    add_files("third_party_module/stdmodules/*.cppm")
    add_includedirs("third_party_module/stdmodules")
    set_languages("c++20")

target("asio")
    set_kind("static")
    add_files("third_party_module/asio/asio.cppm")
    add_includedirs("third_party_module/asio")
    set_languages("c++20")

target("async_simple")
    set_kind("shared")
    add_files("async_simple_module/*.cppm",
              "async_simple_module/coro/*.cppm",
              "async_simple_module/executors/*.cppm",
              "async_simple_module/uthread/*.cppm",
              "async_simple_module/uthread/internal/*.cppm",
              "async_simple_module/util/*.cppm")
    add_deps("std")
    set_languages("c++20")

Both the asio module and std module get compiled correctly. But I received the following message when I try to compile async_simple:

error: On llvm <= 16 standard C++ modules are not supported ;
                           they can be emulated through clang modules and supported only on libc++ ;
                           please add -stdlib=libc++ cxx flag or disable strict mode

The compiler I am using is a source-built clang:

clang version 17.0.0 (git@github.com:llvm/llvm-project.git b40ccea72bad741b0451cb1ff968de4a5f79dd8f)
Target: x86_64-unknown-linux-gnu
Thread model: posix

So xmake didn't detect the version of the compiler correctly. Also it looks not good to prevent users to provide a std module.

Expected Behavior

The target async_simple get compiled correctly.

Project Configuration

No response

Additional Information and Error Logs

No response

waruqi commented 1 year ago

Currently it does not use the llvm/clang version to determine std modules support, it is automatically detected from the json file (p1689) dumped by clang-scan-deps. We will improve it later. @Arthapz

waruqi commented 1 year ago

I have improved clang detection for std modules, but it does not find std module on my machine (clang 17.0).

https://github.com/xmake-io/xmake/pull/3255

please run xmake update -s github:xmake-io/xmake#stdmodules to test it.

ruki-2:stdmodules ruki$ xmake f --toolchain=llvm --sdk=~/projects/working/stdmodules/llvm-project
/llvm/build/ -c; xmake -rv
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
checking for SDK version of Xcode for macosx (x86_64) ... 13.0
[  0%]: generating.module.deps test/test.cpp
checking for /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps
... ok
checking for flags (-fvisibility-inlines-hidden) ... ok
checking for flags (-O3) ... ok
checking for flags (-std=c++23) ... no
checking for flags (-std=c++2b) ... ok
checking for flags (-fmodules) ... ok
checking for flags (-DNDEBUG) ... ok
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c test/tes
t.cpp -o build/.objs/stdmodules/macosx/x86_64/release/test/test.cpp.o -Qunused-arguments -m64 -is
ysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX
13.0.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++2b -fmodules -fno-implicit-m
odule-maps -DNDEBUG
checking for flags (clang_modules_cache_path) ... ok
[  0%]: generating.module.deps src/my_module.cpp
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c src/my_m
odule.cpp -o build/.objs/mod/macosx/x86_64/release/src/my_module.cpp.o -Qunused-arguments -m64 -i
sysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOS
X13.0.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++2b -fmodules -fno-implicit-
module-maps -DNDEBUG
[  0%]: generating.module.deps src/my_module.mpp
checking for flags (-stdlib=libc++) ... ok
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c src/my_m
odule.mpp -o build/.objs/mod/macosx/x86_64/release/src/my_module.mpp.o -Qunused-arguments -m64 -i
sysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOS
X13.0.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++2b -fmodules -fno-implicit-
module-maps -stdlib=libc++ -DNDEBUG
checking for flags (clang_module_output) ... ok
[  7%]: compiling.module.release my_module
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -c -x c++-module -fmodu
le-output=build/.gens/mod/macosx/x86_64/release/rules/modules/cache/81262397/my_module.pcm -Qunus
ed-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/
Developer/SDKs/MacOSX13.0.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++2b -fmo
dules -fno-implicit-module-maps -stdlib=libc++ -DNDEBUG -fmodules-cache-path=build/.gens/mod/maco
sx/x86_64/release/rules/modules/cache -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/C
ontents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fvisibility=hidden -fv
isibility-inlines-hidden -O3 -std=c++2b -fmodules -fno-implicit-module-maps -stdlib=libc++ -DNDEB
UG -fmodules-cache-path=build/.gens/mod/macosx/x86_64/release/rules/modules/cache -o build/.objs/
mod/macosx/x86_64/release/src/my_module.mpp.o build/.gens/mod/macosx/x86_64/release/rules/modules
/cache/81262397/my_module.pcm
checking for flags (-fdiagnostics-color=always) ... ok
error: src/my_module.mpp:3:8: fatal error: module 'std' not found
import std;
~~~~~~~^~~
1 error generated.

some related patches: https://github.com/xmake-io/xmake/pull/3255

waruqi commented 1 year ago

I am trying to compile https://github.com/alibaba/async_simple/tree/CXX20Modules with xmake.

I tried it on stdmodules branch. The previous issue has been fixed, but libaio.h not found. Maybe it should work on linux, you can test it.

ruki-2:async_simple ruki$ xmake f --toolchain=llvm --sdk=~/projects/working/stdmodules/llvm-proje
ct/llvm/build/ -c; xmake -rv
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
checking for SDK version of Xcode for macosx (x86_64) ... 13.0
[  0%]: generating.module.deps async_simple_module/Executor.cppm
checking for /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps
... ok
checking for flags (-O3) ... ok
checking for flags (-std=c++20) ... ok
checking for flags (-fmodules) ... ok
checking for flags (-DNDEBUG) ... ok
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/Executor.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_module/E
xecutor.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/Platf
orms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-implicit-m
odule-maps -DNDEBUG
[  0%]: generating.module.deps async_simple_module/Try.cppm
checking for flags (-stdlib=libc++) ... ok
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/Try.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_module/Try.cp
pm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOS
X.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-implicit-module-maps
 -stdlib=libc++ -DNDEBUG
[  0%]: generating.module.deps async_simple_module/async_simple.cppm
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/async_simple.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_modu
le/async_simple.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Develop
er/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-im
plicit-module-maps -stdlib=libc++ -DNDEBUG
[  0%]: generating.module.deps async_simple_module/Traits.cppm
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/Traits.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_module/Tra
its.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms
/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-implicit-modul
e-maps -stdlib=libc++ -DNDEBUG
[  0%]: generating.module.deps async_simple_module/MoveWrapper.cppm
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/MoveWrapper.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_modul
e/MoveWrapper.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer
/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-impl
icit-module-maps -stdlib=libc++ -DNDEBUG
[  0%]: generating.module.deps async_simple_module/Common.cppm
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/Common.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_module/Com
mon.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms
/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-implicit-modul
e-maps -stdlib=libc++ -DNDEBUG
[  0%]: generating.module.deps async_simple_module/LocalState.cppm
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/LocalState.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_module
/LocalState.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/P
latforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-implic
it-module-maps -stdlib=libc++ -DNDEBUG
[  0%]: generating.module.deps async_simple_module/IOExecutor.cppm
/Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang-scan-deps --format=p168
9 -- /Users/ruki/projects/working/stdmodules/llvm-project/llvm/build/bin/clang -x c++ -c async_si
mple_module/IOExecutor.cppm -o build/.objs/async_simple/macosx/x86_64/release/async_simple_module
/IOExecutor.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/P
latforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -fPIC -O3 -std=c++20 -fmodules -fno-implic
it-module-maps -stdlib=libc++ -DNDEBUG
error: Error while scanning dependencies for async_simple_module/IOExecutor.cppm:
async_simple_module/IOExecutor.cppm:18:10: fatal error: 'libaio.h' file not found

ChuanqiXu9 commented 1 year ago

The stdmodules branch solves the problem. But it introduces the problem I didn't meet in trunk:

It will preprocess sources without language linkage. So the result is incorrect due to incorrect predefined macros.
It will generate BMI files in the form std:queue.pcm but the command line is `-fmodule-file=/path/std-queue.pcm.

BTW, it prefer to use the form -fmodule-file=<module-name>=<BMI-path> than -fmodule-file=<BMI-path>. See https://clang.llvm.org/docs/StandardCPlusPlusModules.html#how-to-specify-the-dependent-bmis.

Also I saw -fmodules -fno-implicit-module-maps in the command line. But it enables different modules than standard c++ modules. So it looks unnecessary to enable this by default to me.

Arthapz commented 1 year ago

It will preprocess sources without language linkage. So the result is incorrect due to incorrect predefined macros.

yeah i removed the -std=c++20 / -std=c++2b from command line because of https://github.com/llvm/llvm-project/issues/59287 but i see this issue is fixed, i can reenable them to llvm >= 16

BTW, it prefer to use the form -fmodule-file== than -fmodule-file=. See https://clang.llvm.org/docs/StandardCPlusPlusModules.html#how-to-specify-the-dependent-bmis.

i already made this change on my module dev branch after you LLVM commit was pushed, i'll PR it soon

Also I saw -fmodules -fno-implicit-module-maps in the command line. But it enables different modules than standard c++ modules. So it looks unnecessary to enable this by default to me.

is it -fmodules is not needed for C++20 modules ?

Arthapz commented 1 year ago

when removing -fno-implicit-module-maps i'm getting llvm crash with libc++ :D

"/home/arthapz/xmake/tests/projects/c++/modules/hello/test.lua"
>> testing tests/projects/c++/modules/hello ...
>>     running test_main ...
checking for gcc ... /usr/bin/gcc
checking for platform ... linux
checking for architecture ... x86_64
[  0%]: generating.module.deps src/main.cpp
[  0%]: generating.module.deps src/hello.mpp
[ 16%]: compiling.module.release hello
[ 50%]: compiling.release src/main.cpp
[ 66%]: linking.release hello
[100%]: build ok!
checking for clang ... /usr/bin/clang
checking for platform ... linux
checking for architecture ... x86_64
[  0%]: generating.module.deps src/main.cpp
[  0%]: generating.module.deps src/hello.mpp
[ 14%]: compiling.module.release hello
[ 57%]: compiling.release src/main.cpp
[ 71%]: linking.release hello
[100%]: build ok!
checking for platform ... linux
checking for architecture ... x86_64
[  0%]: generating.module.deps src/main.cpp
[  0%]: generating.module.deps src/hello.mpp
error: LLVM ERROR: unexpected call to lookupModuleOutput
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x00007fa1dec8f313 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/bin/../lib/libLLVM-17.so+0xe8f313)
 #1 0x00007fa1dec8c6ef llvm::sys::RunSignalHandlers() (/usr/bin/../lib/libLLVM-17.so+0xe8c6ef)
 #2 0x00007fa1dec8c83d (/usr/bin/../lib/libLLVM-17.so+0xe8c83d)
 #3 0x00007fa1dd842cc0 __restore_rt libc_sigaction.c:0:0
 #4 0x00007fa1dd8bbc6c __pthread_kill_implementation pthread_kill.c:0:0
 #5 0x00007fa1dd86a1c6 gsignal (/usr/bin/../lib/libc.so.6+0x4b1c6)
 #6 0x00007fa1dd841875 abort (/usr/bin/../lib/libc.so.6+0x22875)
 #7 0x00007fa1de9d1235 (/usr/bin/../lib/libLLVM-17.so+0xbd1235)
 #8 0x00007fa1debadb8e (/usr/bin/../lib/libLLVM-17.so+0xdadb8e)
 #9 0x00007fa1e95f5f9c (/usr/bin/../lib/libclang-cpp.so.17+0x2df5f9c)
#10 0x00007fa1e96017b0 clang::tooling::dependencies::ModuleDepCollector::addOutputPaths(clang::CompilerInvocation&, clang::tooling::dependencies::ModuleDeps&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e017b0)
#11 0x00007fa1e9609509 clang::tooling::dependencies::ModuleDepCollectorPP::handleTopLevelModule(clang::Module const*) (/usr/bin/../lib/libclang-cpp.so.17+0x2e09509)
#12 0x00007fa1e9609df2 clang::tooling::dependencies::ModuleDepCollectorPP::addModuleDep(clang::Module const*, clang::tooling::dependencies::ModuleDeps&, llvm::DenseSet<clang::Module const*, llvm::DenseMapInfo<clang::Module const*, void>>&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e09df2)
#13 0x00007fa1e960a08b clang::tooling::dependencies::ModuleDepCollectorPP::addAllSubmoduleDeps(clang::Module const*, clang::tooling::dependencies::ModuleDeps&, llvm::DenseSet<clang::Module const*, llvm::DenseMapInfo<clang::Module const*, void>>&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e0a08b)
#14 0x00007fa1e960a23d clang::tooling::dependencies::ModuleDepCollectorPP::addAllSubmoduleDeps(clang::Module const*, clang::tooling::dependencies::ModuleDeps&, llvm::DenseSet<clang::Module const*, llvm::DenseMapInfo<clang::Module const*, void>>&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e0a23d)
#15 0x00007fa1e9609460 clang::tooling::dependencies::ModuleDepCollectorPP::handleTopLevelModule(clang::Module const*) (/usr/bin/../lib/libclang-cpp.so.17+0x2e09460)
#16 0x00007fa1e960a64a clang::tooling::dependencies::ModuleDepCollectorPP::EndOfMainFile() (/usr/bin/../lib/libclang-cpp.so.17+0x2e0a64a)
#17 0x00007fa1e94117f5 clang::FrontendAction::EndSourceFile() (/usr/bin/../lib/libclang-cpp.so.17+0x2c117f5)
#18 0x00007fa1e938b54c clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/bin/../lib/libclang-cpp.so.17+0x2b8b54c)
#19 0x00007fa1e95f2953 (/usr/bin/../lib/libclang-cpp.so.17+0x2df2953)
#20 0x00007fa1e9673f88 clang::tooling::ToolInvocation::run() (/usr/bin/../lib/libclang-cpp.so.17+0x2e73f88)
#21 0x00007fa1e95f495e clang::tooling::dependencies::DependencyScanningWorker::computeDependencies(llvm::StringRef, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, clang::tooling::dependencies::DependencyConsumer&, clang::DiagnosticConsumer&, std::optional<llvm::StringRef>) (/usr/bin/../lib/libclang-cpp.so.17+0x2df495e)
#22 0x00007fa1e95f5b2e clang::tooling::dependencies::DependencyScanningWorker::computeDependencies(llvm::StringRef, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, clang::tooling::dependencies::DependencyConsumer&, std::optional<llvm::StringRef>) (/usr/bin/../lib/libclang-cpp.so.17+0x2df5b2e)
#23 0x00007fa1e95f844b clang::tooling::dependencies::DependencyScanningTool::getP1689ModuleDependencyFile(clang::tooling::CompileCommand const&, llvm::StringRef) (/usr/bin/../lib/libclang-cpp.so.17+0x2df844b)
#24 0x0000556fa3467a3e (/usr/bin/clang-scan-deps+0x1ba3e)
#25 0x0000556fa3457b72 (/usr/bin/clang-scan-deps+0xbb72)
#26 0x00007fa1dec1a964 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/usr/bin/../lib/libLLVM-17.so+0xe1a964)
#27 0x00007fa1dec1b35a (/usr/bin/../lib/libLLVM-17.so+0xe1b35a)
#28 0x00007fa1dd8b9e74 start_thread pthread_create.c:0:0
#29 0x00007fa1dd854130 __GI___clone3 (/usr/bin/../lib/libc.so.6+0x35130)

>>     test failed: ./xmake/core/sandbox/modules/os.lua:332: exec(xmake -r) failed(255)

Arthapz commented 1 year ago

ah it's from clang-scan-deps

[  0%]: generating.module.deps src/hello.mpp
/usr/bin/clang-scan-deps --format=p1689 -- /usr/bin/clang -x c++ -c src/hello.mpp -o build/.objs/hello/linux/x86_64/release/src/hello.mpp.o -Qunused-arguments -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -fmodules -stdlib=libc++ -DNDEBUG
error: LLVM ERROR: unexpected call to lookupModuleOutput
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x00007fd00be8f313 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/bin/../lib/libLLVM-17.so+0xe8f313)
 #1 0x00007fd00be8c6ef llvm::sys::RunSignalHandlers() (/usr/bin/../lib/libLLVM-17.so+0xe8c6ef)
 #2 0x00007fd00be8c83d (/usr/bin/../lib/libLLVM-17.so+0xe8c83d)
 #3 0x00007fd00aa42cc0 __restore_rt libc_sigaction.c:0:0
 #4 0x00007fd00aabbc6c __pthread_kill_implementation pthread_kill.c:0:0
 #5 0x00007fd00aa6a1c6 gsignal (/usr/bin/../lib/libc.so.6+0x4b1c6)
 #6 0x00007fd00aa41875 abort (/usr/bin/../lib/libc.so.6+0x22875)
 #7 0x00007fd00bbd1235 (/usr/bin/../lib/libLLVM-17.so+0xbd1235)
 #8 0x00007fd00bdadb8e (/usr/bin/../lib/libLLVM-17.so+0xdadb8e)
 #9 0x00007fd0167f5f9c (/usr/bin/../lib/libclang-cpp.so.17+0x2df5f9c)
#10 0x00007fd0168017b0 clang::tooling::dependencies::ModuleDepCollector::addOutputPaths(clang::CompilerInvocation&, clang::tooling::dependencies::ModuleDeps&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e017b0)
#11 0x00007fd016809509 clang::tooling::dependencies::ModuleDepCollectorPP::handleTopLevelModule(clang::Module const*) (/usr/bin/../lib/libclang-cpp.so.17+0x2e09509)
#12 0x00007fd016809df2 clang::tooling::dependencies::ModuleDepCollectorPP::addModuleDep(clang::Module const*, clang::tooling::dependencies::ModuleDeps&, llvm::DenseSet<clang::Module const*, llvm::DenseMapInfo<clang::Module const*, void>>&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e09df2)
#13 0x00007fd01680a08b clang::tooling::dependencies::ModuleDepCollectorPP::addAllSubmoduleDeps(clang::Module const*, clang::tooling::dependencies::ModuleDeps&, llvm::DenseSet<clang::Module const*, llvm::DenseMapInfo<clang::Module const*, void>>&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e0a08b)
#14 0x00007fd01680a23d clang::tooling::dependencies::ModuleDepCollectorPP::addAllSubmoduleDeps(clang::Module const*, clang::tooling::dependencies::ModuleDeps&, llvm::DenseSet<clang::Module const*, llvm::DenseMapInfo<clang::Module const*, void>>&) (/usr/bin/../lib/libclang-cpp.so.17+0x2e0a23d)
#15 0x00007fd016809460 clang::tooling::dependencies::ModuleDepCollectorPP::handleTopLevelModule(clang::Module const*) (/usr/bin/../lib/libclang-cpp.so.17+0x2e09460)
#16 0x00007fd01680a64a clang::tooling::dependencies::ModuleDepCollectorPP::EndOfMainFile() (/usr/bin/../lib/libclang-cpp.so.17+0x2e0a64a)
#17 0x00007fd0166117f5 clang::FrontendAction::EndSourceFile() (/usr/bin/../lib/libclang-cpp.so.17+0x2c117f5)
#18 0x00007fd01658b54c clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/bin/../lib/libclang-cpp.so.17+0x2b8b54c)
#19 0x00007fd0167f2953 (/usr/bin/../lib/libclang-cpp.so.17+0x2df2953)
#20 0x00007fd016873f88 clang::tooling::ToolInvocation::run() (/usr/bin/../lib/libclang-cpp.so.17+0x2e73f88)
#21 0x00007fd0167f495e clang::tooling::dependencies::DependencyScanningWorker::computeDependencies(llvm::StringRef, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, clang::tooling::dependencies::DependencyConsumer&, clang::DiagnosticConsumer&, std::optional<llvm::StringRef>) (/usr/bin/../lib/libclang-cpp.so.17+0x2df495e)
#22 0x00007fd0167f5b2e clang::tooling::dependencies::DependencyScanningWorker::computeDependencies(llvm::StringRef, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, clang::tooling::dependencies::DependencyConsumer&, std::optional<llvm::StringRef>) (/usr/bin/../lib/libclang-cpp.so.17+0x2df5b2e)
#23 0x00007fd0167f844b clang::tooling::dependencies::DependencyScanningTool::getP1689ModuleDependencyFile(clang::tooling::CompileCommand const&, llvm::StringRef) (/usr/bin/../lib/libclang-cpp.so.17+0x2df844b)
#24 0x0000563629562a3e (/usr/bin/clang-scan-deps+0x1ba3e)
#25 0x0000563629552b72 (/usr/bin/clang-scan-deps+0xbb72)
#26 0x00007fd00be1a964 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/usr/bin/../lib/libLLVM-17.so+0xe1a964)
#27 0x00007fd00be1b35a (/usr/bin/../lib/libLLVM-17.so+0xe1b35a)
#28 0x00007fd00aab9e74 start_thread pthread_create.c:0:0
#29 0x00007fd00aa54130 __GI___clone3 (/usr/bin/../lib/libc.so.6+0x35130)

this llvm was built just after you revert the 4/4 clang-scan-deps commit, so maybe it is the problem

waruqi commented 1 year ago

i already made this change on my module dev branch after you LLVM commit was pushed, i'll PR it soon

@Arthapz you can push to stdmodules branch first or merge some patches from this branch.

Arthapz commented 1 year ago

is it -fmodules is not needed for C++20 modules ?

ok it's not needed

"warning: the '-fmodules-ts' flag is deprecated and it will be removed in Clang 17; use '-std=c++20' or higher to use standard C++ modules instead [-Wdeprecated-module-ts]" what is the first llvm version wich deprecate -fmodules-ts ?

Arthapz commented 1 year ago

@waruqi it's seems that our header unit detection was broken, enabling -fmodules prevent to really check if header units are supported

i removed it

function has_headerunitsupport(target)
    local support_headerunits = _g.support_headerunits
    if support_headerunits == nil then
        local compinst = target:compiler("cxx")
        local _, modulestsflag, withoutflag = get_modulesflag(target)
        modulestsflag = withoutflag and "" or modulestsflag
        if compinst:has_flags(modulestsflag .. " -std=c++20 -x c++-user-header", "cxxflags", {flagskey = "clang_user_header_unit_support", tryrun = true}) and
           compinst:has_flags(modulestsflag .. " -std=c++20 -x c++-system-header", "cxxflags", {flagskey = "clang_system_header_unit_support", tryrun = true}) then
            support_headerunits = true
        end
        _g.support_headerunits = support_headerunits or false
    end
    return support_headerunits or nil
end

and i get

error: non-inline external definitions are not permitted in C++ header units
int main(int argc, char** argv)

because has_flags compile a regular cpp file is there a way to pass a custom snippet to has_flags ?

like

inline int foo() { return 0; }

ChuanqiXu9 commented 1 year ago

is it -fmodules is not needed for C++20 modules ?

Yes. -fmodules is not needed for C++20 modules. About 3 years ago, there were ideas to implement header units by the existing clang modules. So you may find the related thing. But in the last year we changed the mind. Now -fmodules should be completely unrelated with C++20 Modules.

this llvm was built just after you revert the 4/4 clang-scan-deps commit, so maybe it is the problem

I'll take a look. And I had recommitted the clang-scan-deps patches. Maybe you can try again.

what is the first llvm version wich deprecate -fmodules-ts ?

llvm16 will deprecated -fmodules-ts and llvm17 will remove it.

BTW, we are still discussing how to support header units between compiler and build systems. Since the build systems lack a good method to detect whether the header units are the same. For example, the following two header units may be two different header units, while it is also possible that they are the same header units:

// CC -std=c++20 -DMacro ....
import "a.h";

// CC -std=c++20 ....
import "a.h"

Since the build systems are hard to know whether the macro Macro affects "a.h". And this is the reason why CMake didn't support header units in his experimental version, FYI.

waruqi commented 1 year ago

@waruqi it's seems that our header unit detection was broken, enabling -fmodules prevent to really check if header units are supported

i removed it

function has_headerunitsupport(target)
    local support_headerunits = _g.support_headerunits
    if support_headerunits == nil then
        local compinst = target:compiler("cxx")
        local _, modulestsflag, withoutflag = get_modulesflag(target)
        modulestsflag = withoutflag and "" or modulestsflag
        if compinst:has_flags(modulestsflag .. " -std=c++20 -x c++-user-header", "cxxflags", {flagskey = "clang_user_header_unit_support", tryrun = true}) and
           compinst:has_flags(modulestsflag .. " -std=c++20 -x c++-system-header", "cxxflags", {flagskey = "clang_system_header_unit_support", tryrun = true}) then
            support_headerunits = true
        end
        _g.support_headerunits = support_headerunits or false
    end
    return support_headerunits or nil
end

and i get

error: non-inline external definitions are not permitted in C++ header units
int main(int argc, char** argv)

because has_flags compile a regular cpp file is there a way to pass a custom snippet to has_flags ?

like

inline int foo() { return 0; }

I have improved it to pass a code snippet to has_flags on stdmodules branch, you can try

        if compinst:has_flags(modulestsflag .. " -std=c++20 -x c++-user-header", "cxxflags", {
           snippet = "inline int foo() { return 0; }",
           flagskey = "clang_user_header_unit_support", tryrun = true}) then
        end

Arthapz commented 1 year ago

@ChuanqiXu9 with my PR and with this xmake.lua i got to build until

add_rules("mode.debug", "mode.release")

target("std")
    set_kind("static")
    add_files("third_party_module/stdmodules/*.cppm")
    add_includedirs("third_party_module/stdmodules")
    set_languages("c++20")
    set_policy("build.c++.modules", true)

target("asio")
    set_kind("static")
    add_files("third_party_module/asio/asio.cppm")
    add_includedirs("third_party_module/asio")
    set_languages("c++20")
    set_policy("build.c++.modules", true)

target("async_simple")
    set_kind("shared")
    add_files("async_simple_module/*.cppm",
              "async_simple_module/coro/*.cppm",
              "async_simple_module/executors/*.cppm",
              "async_simple_module/uthread/*.cppm",
              "async_simple_module/uthread/internal/*.cppm",
              "async_simple_module/util/*.cppm")
    add_deps("std")
    set_languages("c++20")
    set_policy("build.c++.modules", true)

i got to build until

[  7%]: compiling.module.release std:tuple
/usr/bin/clang -c -x c++-module -fmodule-output=build/.gens/std/linux/x86_64/release/rules/modules/cache/b8e837a8/std:tuple.pcm -Qunused-arguments -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -Ithird_party_module/stdmodules -stdlib=libc++ -DNDEBUG -fmodules-cache-path=build/.gens/std/linux/x86_64/release/rules/modules/cache -Qunused-arguments -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -Ithird_party_module/stdmodules -stdlib=libc++ -DNDEBUG -fmodules-cache-path=build/.gens/std/linux/x86_64/release/rules/modules/cache -o build/.objs/std/linux/x86_64/release/third_party_module/stdmodules/tuple.cppm.o third_party_module/stdmodules/tuple.cppm
error: @programdir/modules/private/async/runjobs.lua:256: @programdir/rules/c++/modules/modules_support/clang.lua:258: @programdir/modules/core/tools/gcc.lua:713: In file included from third_party_module/stdmodules/tuple.cppm:3:
third_party_module/stdmodules/tuple.cppm:9:16: error: using declaration referring to 'ignore' with internal linkage cannot be exported
    using std::ignore;
               ^
/usr/bin/../include/c++/v1/tuple:1505:39: note: target of using declaration
  constexpr __ignore_t<unsigned char> ignore = __ignore_t<unsigned char>();
                                      ^
1 error generated.

replacing the line with

inline constexpr decltype(std::ignore) ignore = std::ignore;

get me to

[ 55%]: compiling.module.release async_simple:coro.Task
error: @programdir/modules/private/async/runjobs.lua:256: @programdir/rules/c++/modules/modules_support/clang.lua:258: @programdir/modules/core/tools/gcc.lua:713: error: default visibility for functions and variables [-fvisibility] differs in PCH file vs. current file
error: module file build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]
async_simple_module/coro/Task.cppm:38:20: error: use of undeclared identifier 'std'
    using Handle = std::coroutine_handle<promise_type>;
                   ^
async_simple_module/coro/Task.cppm:42:13: error: use of undeclared identifier 'std'
            std::coroutine_handle<> continuation) const noexcept {
            ^
async_simple_module/coro/Task.cppm:54:10: error: unknown type name 'Handle'
    Task(Handle coro) : _coro(coro), _hasValue(false) {
         ^
async_simple_module/coro/Task.cppm:90:5: error: unknown type name 'Handle'
    Handle _coro;
    ^
async_simple_module/coro/Task.cppm:45:20: error: use of undeclared identifier 'std'
            return std::move(task->stealValue());
                   ^
async_simple_module/coro/Task.cppm:63:17: error: use of undeclared identifier 'std'
        : _coro(std::exchange(other._coro, nullptr)),
                ^
async_simple_module/coro/Task.cppm:64:21: error: use of undeclared identifier 'std'
          _hasValue(std::exchange(other._hasValue, false)) {
                    ^
async_simple_module/coro/Task.cppm:65:14: error: use of undeclared identifier 'std'
        new (std::addressof(_value)) T(std::move(other._value));
             ^
async_simple_module/coro/Task.cppm:80:14: error: use of undeclared identifier 'std'
        new (std::addressof(_value)) T(std::forward<C>(v));
             ^
async_simple_module/coro/Task.cppm:86:16: error: use of undeclared identifier 'std'
        return std::move(_value);
               ^
async_simple_module/coro/Task.cppm:104:20: error: use of undeclared identifier 'std'
    using Handle = std::coroutine_handle<promise_type>;
                   ^
async_simple_module/coro/Task.cppm:108:13: error: use of undeclared identifier 'std'
            std::coroutine_handle<> continuation) const noexcept {
            ^
async_simple_module/coro/Task.cppm:130:15: error: use of undeclared identifier 'std'
    constexpr std::suspend_never initial_suspend() const
              ^
async_simple_module/coro/Task.cppm:134:15: error: use of undeclared identifier 'std'
    constexpr std::suspend_never final_suspend() const noexcept {
              ^
async_simple_module/coro/Task.cppm:139:24: error: use of undeclared identifier 'std'
    void* operator new(std::size_t sz) noexcept { return ::operator new(sz); }
                       ^
async_simple_module/coro/Task.cppm:140:35: error: use of undeclared identifier 'std'
    void operator delete(void* p, std::size_t sz) noexcept {
                                  ^
async_simple_module/coro/Task.cppm:147:90: error: use of undeclared identifier 'std'
    inline __attribute__((__always_inline__))  void return_value(T&& v) { task->setValue(std::move(v)); }

the commandline is

/usr/bin/clang -c -x c++-module -fmodule-output=build/.gens/async_simple/linux/x86_64/release/rules/modules/cache/53a3c03f/async_simple-coro.Task.pcm -fmodule-file=std=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std.pcm -fmodule-file=std:numeric=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-numeric.pcm -fmodule-file=std:string=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-string.pcm -fmodule-file=std:cstdio=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdio.pcm -fmodule-file=std:queue=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-queue.pcm -fmodule-file=std:exception=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-exception.pcm -fmodule-file=std:unordered_map=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-unordered_map.pcm -fmodule-file=std:condition_variable=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-condition_variable.pcm -fmodule-file=std:cstdint=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdint.pcm -fmodule-file=std:thread=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-thread.pcm -fmodule-file=std:coroutine=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-coroutine.pcm -fmodule-file=std:initializer_list=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-initializer_list.pcm -fmodule-file=std:map=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-map.pcm -fmodule-file=std:functional=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-functional.pcm -fmodule-file=std:utility=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-utility.pcm -fmodule-file=std:fstream=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-fstream.pcm -fmodule-file=std:cstdlib=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdlib.pcm -fmodule-file=std:iterator=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iterator.pcm -fmodule-file=std:limits=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-limits.pcm -fmodule-file=std:stdexcept=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-stdexcept.pcm -fmodule-file=std:list=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-list.pcm -fmodule-file=std:type_traits=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-type_traits.pcm -fmodule-file=std:memory=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-memory.pcm -fmodule-file=std:chrono=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-chrono.pcm -fmodule-file=std:algorithm=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-algorithm.pcm -fmodule-file=std:atomic=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm -fmodule-file=std:array=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-array.pcm -fmodule-file=std:mutex=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-mutex.pcm -fmodule-file=std:random=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-random.pcm -fmodule-file=std:iostream=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iostream.pcm -fmodule-file=std:tuple=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-tuple.pcm -fmodule-file=std:vector=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-vector.pcm -Qunused-arguments -m64 -fPIC -O3 -std=c++20 -stdlib=libc++ -DNDEBUG -fmodules-cache-path=build/.gens/async_simple/linux/x86_64/release/rules/modules/cache -fmodule-file=std=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std.pcm -fmodule-file=std:numeric=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-numeric.pcm -fmodule-file=std:string=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-string.pcm -fmodule-file=std:cstdio=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdio.pcm -fmodule-file=std:queue=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-queue.pcm -fmodule-file=std:exception=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-exception.pcm -fmodule-file=std:unordered_map=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-unordered_map.pcm -fmodule-file=std:condition_variable=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-condition_variable.pcm -fmodule-file=std:cstdint=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdint.pcm -fmodule-file=std:thread=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-thread.pcm -fmodule-file=std:coroutine=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-coroutine.pcm -fmodule-file=std:initializer_list=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-initializer_list.pcm -fmodule-file=std:map=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-map.pcm -fmodule-file=std:functional=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-functional.pcm -fmodule-file=std:utility=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-utility.pcm -fmodule-file=std:fstream=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-fstream.pcm -fmodule-file=std:cstdlib=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdlib.pcm -fmodule-file=std:iterator=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iterator.pcm -fmodule-file=std:limits=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-limits.pcm -fmodule-file=std:stdexcept=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-stdexcept.pcm -fmodule-file=std:list=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-list.pcm -fmodule-file=std:type_traits=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-type_traits.pcm -fmodule-file=std:memory=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-memory.pcm -fmodule-file=std:chrono=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-chrono.pcm -fmodule-file=std:algorithm=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-algorithm.pcm -fmodule-file=std:atomic=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm -fmodule-file=std:array=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-array.pcm -fmodule-file=std:mutex=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-mutex.pcm -fmodule-file=std:random=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-random.pcm -fmodule-file=std:iostream=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iostream.pcm -fmodule-file=std:tuple=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-tuple.pcm -fmodule-file=std:vector=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-vector.pcm -o build/.objs/async_simple/linux/x86_64/release/async_simple_module/coro/Task.cppm.o async_simple_module/coro/Task.cppm

wich is the incorrect language linkage you mentioned ?

waruqi commented 1 year ago

I have merged this patch https://github.com/xmake-io/xmake/pull/3367

@ChuanqiXu9 you can update to the latest commit from stdmodules branch and test it again.

xmake update -s github:xmake-io/xmake#stdmodules

ChuanqiXu9 commented 1 year ago

@Arthapz This first issue you met is about the different implementation for STL and your workaround looks not bad. And your second problem looks like due to the inconsistent options.

@waruqi Now it can't find the imported BMIs. When I compile std.cppm, it tells that it can't find the partitions. The command line is:

/home/chuanqi.xcq/llvm-project-for-work/build/bin/clang++ -c -x c++-module -fmodule-output=build/.gens/std/linux/x86_64/release/rules/modules/cache/b8e837a8/std.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-tuple.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-unordered_map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-queue.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-initializer_list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-type_traits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-string.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-thread.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-array.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-mutex.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-condition_variable.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-numeric.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-functional.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-vector.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdio.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iostream.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdlib.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-random.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-utility.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-exception.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iterator.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-chrono.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-algorithm.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-memory.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdint.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-limits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-coroutine.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-stdexcept.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-fstream.pcm -Qunused-arguments -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -Ithird_party_module/stdmodules -DNDEBUG -fmodules-cache-path=build/.gens/std/linux/x86_64/release/rules/modules/cache -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-tuple.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-unordered_map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-queue.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-initializer_list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-type_traits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-string.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-thread.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-array.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-mutex.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-condition_variable.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-numeric.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-functional.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-vector.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdio.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iostream.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdlib.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-random.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-utility.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-exception.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iterator.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-chrono.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-algorithm.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-memory.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdint.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-limits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-coroutine.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-stdexcept.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-fstream.pcm -o build/.objs/std/linux/x86_64/release/third_party_module/stdmodules/std.cppm.o third_party_module/stdmodules/std.cppm

The error message is:

error: fatal error: module file 'build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm' not found: module file not found
1 error generated.

And the directory build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9 doesn't exist indeed. But interestingly, build/.gens/std/linux/x86_64/release/rules/modules/cache/b8e837a8 exists. And its contents are:

algorithm.cppm.json           cstdio.cppm.json            iterator.cppm.json  queue.cppm.json             std:coroutine.pcm    std:fstream.pcm           std:map.pcm        std:string.pcm         string.cppm.json
array.cppm.json               cstdlib.cppm.json           limits.cppm.json    random.cppm.json            std.cppm.json        std:functional.pcm        std:memory.pcm     std:thread.pcm         thread.cppm.json
atomic.cppm.json              exception.cppm.json         list.cppm.json      std:algorithm.pcm           std:cstdint.pcm      std:initializer_list.pcm  std:mutex.pcm      std:tuple.pcm          tuple.cppm.json
chrono.cppm.json              fstream.cppm.json           map.cppm.json       std:array.pcm               std:cstdio.pcm       std:iostream.pcm          std:numeric.pcm    std:type_traits.pcm    type_traits.cppm.json
condition_variable.cppm.json  functional.cppm.json        memory.cppm.json    std:atomic.pcm              std:cstdlib.pcm      std:iterator.pcm          std:queue.pcm      std:unordered_map.pcm  unordered_map.cppm.json
coroutine.cppm.json           initializer_list.cppm.json  mutex.cppm.json     std:chrono.pcm              stdexcept.cppm.json  std:limits.pcm            std:random.pcm     std:utility.pcm        utility.cppm.json
cstdint.cppm.json             iostream.cppm.json          numeric.cppm.json   std:condition_variable.pcm  std:exception.pcm    std:list.pcm              std:stdexcept.pcm  std:vector.pcm         vector.cppm.json

So it looks like it can work if we change std:part.pcm to std-part.pcm and change 684f07d9 to b8e837a8.

waruqi commented 1 year ago

@Arthapz This first issue you met is about the different implementation for STL and your workaround looks not bad. And your second problem looks like due to the inconsistent options.

@waruqi Now it can't find the imported BMIs. When I compile std.cppm, it tells that it can't find the partitions. The command line is:

/home/chuanqi.xcq/llvm-project-for-work/build/bin/clang++ -c -x c++-module -fmodule-output=build/.gens/std/linux/x86_64/release/rules/modules/cache/b8e837a8/std.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-tuple.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-unordered_map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-queue.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-initializer_list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-type_traits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-string.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-thread.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-array.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-mutex.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-condition_variable.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-numeric.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-functional.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-vector.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdio.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iostream.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdlib.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-random.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-utility.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-exception.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iterator.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-chrono.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-algorithm.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-memory.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdint.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-limits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-coroutine.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-stdexcept.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-fstream.pcm -Qunused-arguments -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -Ithird_party_module/stdmodules -DNDEBUG -fmodules-cache-path=build/.gens/std/linux/x86_64/release/rules/modules/cache -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-tuple.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-unordered_map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-queue.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-initializer_list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-type_traits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-map.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-string.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-thread.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-array.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-mutex.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-condition_variable.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-numeric.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-functional.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-list.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-vector.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdio.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iostream.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdlib.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-random.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-utility.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-exception.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-iterator.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-chrono.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-algorithm.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-memory.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-cstdint.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-limits.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-coroutine.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-stdexcept.pcm -fmodule-file=build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-fstream.pcm -o build/.objs/std/linux/x86_64/release/third_party_module/stdmodules/std.cppm.o third_party_module/stdmodules/std.cppm

The error message is:

error: fatal error: module file 'build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std-atomic.pcm' not found: module file not found
1 error generated.

And the directory build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9 doesn't exist indeed. But interestingly, build/.gens/std/linux/x86_64/release/rules/modules/cache/b8e837a8 exists. And its contents are:

algorithm.cppm.json           cstdio.cppm.json            iterator.cppm.json  queue.cppm.json             std:coroutine.pcm    std:fstream.pcm           std:map.pcm        std:string.pcm         string.cppm.json
array.cppm.json               cstdlib.cppm.json           limits.cppm.json    random.cppm.json            std.cppm.json        std:functional.pcm        std:memory.pcm     std:thread.pcm         thread.cppm.json
atomic.cppm.json              exception.cppm.json         list.cppm.json      std:algorithm.pcm           std:cstdint.pcm      std:initializer_list.pcm  std:mutex.pcm      std:tuple.pcm          tuple.cppm.json
chrono.cppm.json              fstream.cppm.json           map.cppm.json       std:array.pcm               std:cstdio.pcm       std:iostream.pcm          std:numeric.pcm    std:type_traits.pcm    type_traits.cppm.json
condition_variable.cppm.json  functional.cppm.json        memory.cppm.json    std:atomic.pcm              std:cstdlib.pcm      std:iterator.pcm          std:queue.pcm      std:unordered_map.pcm  unordered_map.cppm.json
coroutine.cppm.json           initializer_list.cppm.json  mutex.cppm.json     std:chrono.pcm              stdexcept.cppm.json  std:limits.pcm            std:random.pcm     std:utility.pcm        utility.cppm.json
cstdint.cppm.json             iostream.cppm.json          numeric.cppm.json   std:condition_variable.pcm  std:exception.pcm    std:list.pcm              std:stdexcept.pcm  std:vector.pcm         vector.cppm.json

So it looks like it can work if we change std:part.pcm to std-part.pcm and change 684f07d9 to b8e837a8.

It works for me. on my machine:

/data/ruki/async_simple$ ls build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/
std-algorithm.pcm  std-condition_variable.pcm  std-cstdlib.pcm     std-initializer_list.pcm  std-list.pcm    std-numeric.pcm  std-stdexcept.pcm  std-type_traits.pcm
std-array.pcm      std-coroutine.pcm           std-exception.pcm   std-iostream.pcm          std-map.pcm     std.pcm          std-string.pcm     std-unordered_map.pcm
std-atomic.pcm     std-cstdint.pcm             std-fstream.pcm     std-iterator.pcm          std-memory.pcm  std-queue.pcm    std-thread.pcm     std-utility.pcm
std-chrono.pcm     std-cstdio.pcm              std-functional.pcm  std-limits.pcm            std-mutex.pcm   std-random.pcm   std-tuple.pcm      std-vector.pcm
/data/ruki/async_simple$ ls build/.gens/std/linux/x86_64/release/rules/modules/cache/b8e837a8/
algorithm.cppm.json  condition_variable.cppm.json  cstdlib.cppm.json     initializer_list.cppm.json  list.cppm.json    numeric.cppm.json  stdexcept.cppm.json  type_traits.cppm.json
array.cppm.json      coroutine.cppm.json           exception.cppm.json   iostream.cppm.json          map.cppm.json     queue.cppm.json    string.cppm.json     unordered_map.cppm.json
atomic.cppm.json     cstdint.cppm.json             fstream.cppm.json     iterator.cppm.json          memory.cppm.json  random.cppm.json   thread.cppm.json     utility.cppm.json
chrono.cppm.json     cstdio.cppm.json              functional.cppm.json  limits.cppm.json            mutex.cppm.json   std.cppm.json      tuple.cppm.json      vector.cppm.json

Can you try rm -rf build; rm -rf .xmake and rebuild it again?

But I got another error.

s/modules/cache/684f07d9/std-stdexcept.pcm -o build/.objs/async_simple/linux/x86_64/release/async_simple_module/coro/Task.cppm.o async_simple_module/coro/Task.cppm
error: error: default visibility for functions and variables [-fvisibility] differs in PCH file vs. current file
error: module file build/.gens/std/linux/x86_64/release/rules/modules/cache/684f07d9/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]
async_simple_module/Common.cppm:29:11: error: use of undeclared identifier 'std'
    throw std::logic_error(errorMsg);
          ^

ChuanqiXu9 commented 1 year ago

@waruqi it works now after I remove the .xmake dir.

The error you got is the same with @Arthapz mentioned. Due to we failed to load std.pcm actually, the new source file don't know anything in std module.

ChuanqiXu9 commented 1 year ago

The problem goes away after I set target("async_simple") to set_kind("static"). This may be a defect in the compiler side but I can't be pretty sure. I'll try to look at it.

waruqi commented 1 year ago

The error you got is the same with @Arthapz mentioned. Due to we failed to load std.pcm actually, the new source file don't know anything in std module.

In release mode, we set hidden for static/binary targets by default (to optimize binary size), but only for dynamic libraries we disable hidden (to avoid missing export symbol problems).

If there is a dependency between shared and static targets, this can lead to inconsistencies in their visibility configurations. However, it seems that the same visibility configuration is strictly required between compiled modules.

I have fixed it on stdmodules, you can try it again.

or you can also use set_symbols("hidden") or set_symbols("none") to your all targets. it will override the configuration in mode.release rule.

Also, I don't know why clang needs to strictly detect it, gcc doesn't even seem to require visibility to be consistent.

ChuanqiXu9 commented 1 year ago

Thanks. clang performs a very strict consistency check to avoid the very very annoying runtime bugs which is very hard to debug. Although this strategy may be an overkill sometimes.

xmake-io / xmake