Fix Clang module builds using outdated build flags

refi64 commented 3 months ago

_append_requires_flags won't actually regenerate the flags unless opt.build is true, but the callers never set it, so changing module dependencies won't affect the flags until a clean build.

refi64 commented 3 months ago

Note that this might also affect MSVC, but I can't test that and am unfamiliar with their modules implementation.

Arthapz commented 3 months ago

Hello, i'm on vacancy, i'll review it when i come back

Arthapz commented 3 months ago

can u try https://github.com/Arthapz/xmake/tree/fix-module-scanning and confirm that the issue is fixed or not ?

waruqi commented 3 months ago

can u try https://github.com/Arthapz/xmake/tree/fix-module-scanning and confirm that the issue is fixed or not ?

@refi64 Does it work now? and can you provide a reproducible problem project example?

refi64 commented 3 months ago

@Arthapz no, that doesn't seem to fix it.

@waruqi The simplest repro I have is:

Go to tests/projects/c++/modules/hello. Make sure there is no build cache, xmake clean -a.

Create a new module in src/hello2.mpp like:

export module hello2;
export void nothing() {}

Run xmake project -k compile_commands and xmake. These should both succeed.
Add import hello2 to the top of main.cpp.

Run xmake again. Observe that it now fails:

[  0%]: <hello> generating.module.deps src/main.cpp
[ 37%]: <hello> compiling.module.release hello2
[ 75%]: compiling.release src/main.cpp
error: src/main.cpp:1:8: fatal error: module 'hello2' not found
  1 | import hello2;
    | ~~~~~~~^~~~~~
1 error generated.
> in src/main.cpp

Run xmake clean -a to re-clean the build dir.
Running xmake now builds successfully.

Arthapz commented 3 months ago

@Arthapz no, that doesn't seem to fix it.

@waruqi The simplest repro I have is:

* Go to `tests/projects/c++/modules/hello`. Make sure there is no build cache, `xmake clean -a`.

* Create a new module in `src/hello2.mpp` like:
  ```c++
  export module hello2;
  export void nothing() {}
  ```

* Run `xmake project -k compile_commands` and `xmake`. These should both succeed.

* Add `import hello2` to the top of `main.cpp`.

* Run `xmake` again. Observe that it now fails:
  ```c++
  [  0%]: <hello> generating.module.deps src/main.cpp
  [ 37%]: <hello> compiling.module.release hello2
  [ 75%]: compiling.release src/main.cpp
  error: src/main.cpp:1:8: fatal error: module 'hello2' not found
      1 | import hello2;
        | ~~~~~~~^~~~~~
  1 error generated.
    > in src/main.cpp
  ```

* Run `xmake clean -a` to re-clean the build dir.

* Running `xmake` now builds successfully.

i can't reproduce

work

waruqi commented 2 months ago

I was able to reproduce it on my macOS.

and I tried this patch https://github.com/Arthapz/xmake/tree/fix-module-scanning , it still does not work.

$ cd tests/projects/c++/modules/hello
ruki:hello ruki$ rm -rf build/; rm -rf .xmake/
ruki:hello ruki$ echo 'export module hello2;
export void nothing() {}' > src/hello2.mpp
ruki:hello ruki$ xmake f --toolchain=llvm -c
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
ruki:hello ruki$ xmake project -k compile_commands
[  0%]: <hello> generating.module.deps src/main.cpp
[  0%]: <hello> generating.module.deps src/hello2.mpp
[  0%]: <hello> generating.module.deps src/hello.mpp
[  0%]: <hello> generating.module.deps /usr/local/Cellar/llvm/18.1.5/bin/../lib/c++/../../share/libc++/v1/std.c
ppm
[  0%]: <hello> generating.module.deps /usr/local/Cellar/llvm/18.1.5/bin/../lib/c++/../../share/libc++/v1/std.c
ompat.cppm
create ok!
warning: some modules have got culled, use verbose (-v) mode to more informations
ruki:hello ruki$ xmake
[ 42%]: <hello> compiling.module.release hello
[ 71%]: compiling.release src/main.cpp
[ 85%]: linking.release hello
[100%]: build ok, spent 1.384s
warning: some modules have got culled, use verbose (-v) mode to more informations
ruki:hello ruki$ vim ./src/main.cpp # edit and add `import hello2;`
ruki:hello ruki$ cat ./src/main.cpp
import hello2;
import hello;

int main() {
    hello::say("hello module!");
    return 0;
}
ruki:hello ruki$ xmake
[  0%]: <hello> generating.module.deps src/main.cpp
[ 50%]: <hello> compiling.module.release hello2
[ 75%]: compiling.release src/main.cpp
error: src/main.cpp:1:8: fatal error: module 'hello2' not found
    1 | import hello2;
      | ~~~~~~~^~~~~~
1 error generated.
  > in src/main.cpp

waruqi commented 2 months ago

full verbose logs

ruki:hello ruki$ rm -rf .xmake/
ruki:hello ruki$ rm -rf build/
ruki:hello ruki$ xmake f --toolchain=llvm -c -vD
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
checking for SDK version of Xcode for macosx (x86_64) ... 14.0
checking for clang ... /usr/local/Cellar/llvm/18.1.5/bin/clang
checking for the c++ compiler (cxx) ... clang
checking for /usr/local/Cellar/llvm/18.1.5/bin/clang ... ok
checking for flags (-fPIC) ... ok
> clang "-fPIC" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/M
acOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (clang_modules) ... ok
> clang "-fmodules" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platfor
ms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (clang_modules_ts) ... no
> clang "-fmodules-ts" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Plat
forms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checkinfo: @programdir/core/sandbox/modules/os.lua:273: clang: error: unknown argument: '-fmodules-ts'

stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:973]:
    [@programdir/core/sandbox/modules/os.lua:273]: in function 'runv'
    [@programdir/modules/detect/tools/gcc/has_flags.lua:42]:
checking for /usr/local/Cellar/llvm/18.1.5/bin/clang ... ok
configure
{
    theme = default
    ndk_stdcxx = true
    plat = macosx
    network = public
    toolchain = llvm
    buildir = build
    ccache = true
    kind = static
    proxy_pac = pac.lua
    clean = true
    arch = x86_64
    host = macosx
    mode = release
    xcode = /Applications/Xcode.app
}
ruki:hello ruki$ xmake project -k compile_commands -vD
configure
{
    plat = macosx
    network = public
    clean = true
    theme = default
    arch = x86_64
    proxy_pac = pac.lua
    ndk_stdcxx = true
    mode = release
    xcode = /Applications/Xcode.app
    buildir = build
    kind = static
    ccache = true
    toolchain = llvm
    host = macosx
}
checking for flags (clang_print_library_module_manifest_path) ... no
> clang "-print-library-module-manifest-path" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/
Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checkinfo: @programdir/core/sandbox/modules/os.lua:273: clang: error: unknown argument: '-print-library-module-
manifest-path'

stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:973]:
    [@programdir/core/sandbox/modules/os.lua:273]: in function 'runv'
    [@programdir/modules/detect/tools/gcc/has_flags.lua:42]:
checking for flags (-fvisibility-inlines-hidden) ... ok
> clang "-fvisibility-inlines-hidden" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents
/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (-O3) ... ok
> clang "-O3" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/Mac
OSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (-std=c++20) ... ok
> clang "-std=c++20" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platfo
rms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (-DNDEBUG) ... ok
> clang "-DNDEBUG" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platform
s/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
[  0%]: <hello> generating.module.deps src/main.cpp
checking for /usr/local/Cellar/llvm/18.1.5/bin/clang-scan-deps ... ok
/usr/local/Cellar/llvm/18.1.5/bin/clang-scan-deps --format=p1689 -- /usr/local/Cellar/llvm/18.1.5/bin/clang -x
c++ -c src/main.cpp -o build/.objs/hello/macosx/x86_64/release/src/main.cpp.o -Qunused-arguments -m64 -isysroot
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibilit
y=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -DNDEBUG
[  0%]: <hello> generating.module.deps src/hello2.mpp
/usr/local/Cellar/llvm/18.1.5/bin/clang-scan-deps --format=p1689 -- /usr/local/Cellar/llvm/18.1.5/bin/clang -x
c++ -c src/hello2.mpp -o build/.objs/hello/macosx/x86_64/release/src/hello2.mpp.o -Qunused-arguments -m64 -isys
root /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisib
ility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -DNDEBUG
[  0%]: <hello> generating.module.deps src/hello.mpp
/usr/local/Cellar/llvm/18.1.5/bin/clang-scan-deps --format=p1689 -- /usr/local/Cellar/llvm/18.1.5/bin/clang -x
c++ -c src/hello.mpp -o build/.objs/hello/macosx/x86_64/release/src/hello.mpp.o -Qunused-arguments -m64 -isysro
ot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibil
ity=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -DNDEBUG
[  0%]: <hello> generating.module.deps /usr/local/Cellar/llvm/18.1.5/bin/../lib/c++/../../share/libc++/v1/std.c
ppm
/usr/local/Cellar/llvm/18.1.5/bin/clang-scan-deps --format=p1689 -- /usr/local/Cellar/llvm/18.1.5/bin/clang -x
c++ -c /usr/local/Cellar/llvm/18.1.5/bin/../lib/c++/../../share/libc++/v1/std.cppm -o build/.objs/hello/macosx/
x86_64/release/usr/local/Cellar/llvm/18.1.5/bin/__/lib/c++/__/__/share/libc++/v1/std.cppm.o -Qunused-arguments
-m64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.s
dk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -DNDEBUG
[  0%]: <hello> generating.module.deps /usr/local/Cellar/llvm/18.1.5/bin/../lib/c++/../../share/libc++/v1/std.c
ompat.cppm
/usr/local/Cellar/llvm/18.1.5/bin/clang-scan-deps --format=p1689 -- /usr/local/Cellar/llvm/18.1.5/bin/clang -x
c++ -c /usr/local/Cellar/llvm/18.1.5/bin/../lib/c++/../../share/libc++/v1/std.compat.cppm -o build/.objs/hello/
macosx/x86_64/release/usr/local/Cellar/llvm/18.1.5/bin/__/lib/c++/__/__/share/libc++/v1/std.compat.cppm.o -Qunu
sed-arguments -m64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDK
s/MacOSX14.0.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -DNDEBUG
checking for flags (clang_module_file) ... ok
> clang "-fmodule-file=/var/folders/32/w9cz0y_14hs19lkbs6v6_fm80000gn/T/.xmake501/240819/_4A6038DE78C84C108ED19
5C53DC08D20.pcm" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/
MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for flags (clang_module_output) ... ok
> clang "-fmodule-output=" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/
Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
create ok!
warning: some modules have got culled, because it is not consumed by its target nor flagged as a public module
with add_files("xxx.mpp", {public = true})
    hello:
        hello2 -> src/hello2.mpp
ruki:hello ruki$ xmake project -k compile_commands -vD^C
ruki:hello ruki$ xmake -vD
[ 42%]: <hello> compiling.module.release hello
/usr/local/Cellar/llvm/18.1.5/bin/clang -c -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/D
eveloper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibility=hidden -fvisibility-inlines-hidde
n -O3 -std=c++20 -DNDEBUG -x c++-module -fmodule-output=build/.gens/hello/macosx/x86_64/release/rules/bmi/cache
/modules/c83caa2b/hello.pcm -o build/.objs/hello/macosx/x86_64/release/src/hello.mpp.o src/hello.mpp
checking for flags (-fdiagnostics-color=always) ... ok
> clang "-fdiagnostics-color=always" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/
Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
[ 71%]: compiling.release src/main.cpp
/usr/local/Cellar/llvm/18.1.5/bin/clang -c -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/D
eveloper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibility=hidden -fvisibility-inlines-hidde
n -O3 -std=c++20 -DNDEBUG -fmodule-file=hello=build/.gens/hello/macosx/x86_64/release/rules/bmi/cache/modules/c
83caa2b/hello.pcm -o build/.objs/hello/macosx/x86_64/release/src/main.cpp.o src/main.cpp
checking for flags (-MMD -MF) ... ok
> clang "-MMD" "-MF" "/dev/null" "-Qunused-arguments" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Deve
loper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk"
checking for clang++ ... /usr/local/Cellar/llvm/18.1.5/bin/clang++
checking for the linker (ld) ... clang++
checking for /usr/local/Cellar/llvm/18.1.5/bin/clang++ ... ok
checking for flags (-fPIC) ... ok
> clang++ "-fPIC" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Deve
loper/SDKs/MacOSX14.0.sdk" "-m64" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.plat
form/Developer/SDKs/MacOSX14.0.sdk"
[ 85%]: linking.release hello
/usr/local/Cellar/llvm/18.1.5/bin/clang++ -o build/macosx/x86_64/release/hello build/.objs/hello/macosx/x86_64/
release/src/main.cpp.o build/.objs/hello/macosx/x86_64/release/src/hello.mpp.o -m64 -isysroot /Applications/Xco
de.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -Wl,-x -Wl,-dead_strip

build cache stats:
cache directory: build/.build_cache
cache hit rate: 0%
cache hit: 0
cache hit total time: 0.000s
cache miss: 0
cache miss total time: 0.000s
new cached files: 0
remote cache hit: 0
remote new cached files: 0
preprocess failed: 0
compile fallback count: 0
compile total time: 0.000s

[100%]: build ok, spent 1.284s
warning: some modules have got culled, because it is not consumed by its target nor flagged as a public module
with add_files("xxx.mpp", {public = true})
    hello:
        hello2 -> src/hello2.mpp
ruki:hello ruki$ xmake -vD^C
ruki:hello ruki$ vim ./src/main.cpp
ruki:hello ruki$ xmake -vD
[  0%]: <hello> generating.module.deps src/main.cpp
/usr/local/Cellar/llvm/18.1.5/bin/clang-scan-deps --format=p1689 -- /usr/local/Cellar/llvm/18.1.5/bin/clang -x
c++ -c src/main.cpp -o build/.objs/hello/macosx/x86_64/release/src/main.cpp.o -Qunused-arguments -m64 -isysroot
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibilit
y=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -DNDEBUG
[ 50%]: <hello> compiling.module.release hello2
/usr/local/Cellar/llvm/18.1.5/bin/clang -c -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/D
eveloper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibility=hidden -fvisibility-inlines-hidde
n -O3 -std=c++20 -DNDEBUG -x c++-module -fmodule-output=build/.gens/hello/macosx/x86_64/release/rules/bmi/cache
/modules/c83caa2b/hello2.pcm -o build/.objs/hello/macosx/x86_64/release/src/hello2.mpp.o src/hello2.mpp
[ 75%]: compiling.release src/main.cpp
/usr/local/Cellar/llvm/18.1.5/bin/clang -c -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Contents/D
eveloper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -fvisibility=hidden -fvisibility-inlines-hidde
n -O3 -std=c++20 -DNDEBUG -fmodule-file=hello=build/.gens/hello/macosx/x86_64/release/rules/bmi/cache/modules/c
83caa2b/hello.pcm -o build/.objs/hello/macosx/x86_64/release/src/main.cpp.o src/main.cpp
error: @programdir/core/main.lua:329: @programdir/actions/build/main.lua:148: @programdir/modules/async/runjobs
.lua:325: @programdir/modules/private/action/build/object.lua:91: @programdir/modules/core/tools/gcc.lua:916: s
rc/main.cpp:1:8: fatal error: module 'hello2' not found
    1 | import hello2;
      | ~~~~~~~^~~~~~
1 error generated.
stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:973]:
    [@programdir/modules/core/tools/gcc.lua:916]: in function 'catch'
    [@programdir/core/sandbox/modules/try.lua:123]: in function 'try'
    [@programdir/modules/core/tools/gcc.lua:857]:
    [C]: in function 'xpcall'
    [@programdir/core/base/utils.lua:275]:
    [@programdir/core/tool/compiler.lua:278]: in function 'compile'
    [@programdir/modules/private/action/build/object.lua:91]: in function 'script'
    [@programdir/modules/private/action/build/object.lua:122]: in function 'build_object'
    [@programdir/modules/private/action/build/object.lua:147]: in function 'jobfunc'
    [@programdir/modules/async/runjobs.lua:241]:
    [C]: in function 'xpcall'
    [@programdir/core/base/utils.lua:275]: in function 'trycall'
    [@programdir/core/sandbox/modules/try.lua:117]: in function 'try'
    [@programdir/modules/async/runjobs.lua:223]: in function 'cotask'
    [@programdir/core/base/scheduler.lua:406]:

stack traceback:
        [C]: in function 'error'
        @programdir/core/base/os.lua:973: in function 'base/os.raiselevel'
        (...tail calls...)
        @programdir/core/main.lua:329: in upvalue 'cotask'
        @programdir/core/base/scheduler.lua:406: in function <@programdir/core/base/scheduler.lua:399>
ruki:hello ruki$

Arthapz commented 2 months ago

Of it was working under gcc, but not with clang

Arthapz commented 1 month ago

i updated the branch, try now @waruqi @refi64

waruqi commented 1 month ago

thanks, I will try it in these days.

waruqi commented 1 month ago

import hello2;

it still does not work. same errors.

ruki-2:hello ruki$ xmake
[  0%]: <hello> generating.module.deps src/main.cpp
[ 37%]: <hello> compiling.module.release hello2
[ 75%]: compiling.release src/main.cpp
error: src/main.cpp:2:8: fatal error: module 'hello2' not found
    2 | import hello2;
      | ~~~~~~~^~~~~~
1 error generated.
  > in src/main.cpp

you can try https://github.com/xmake-io/xmake/pull/5425#issuecomment-2296764096

Arthapz commented 1 month ago

well i don't get it to reproduce demo

waruqi commented 1 month ago

I can stably reproduce it on both macOS and Linux. llvm 14-18. Follow my above reproduction process.

waruqi commented 1 month ago

@Arthapz I tried this patch. It works fine now. Perhaps I used the wrong branch before. https://github.com/xmake-io/xmake/pull/5686

waruqi commented 1 month ago

@refi64 It should be fixed now. https://github.com/xmake-io/xmake/pull/5686

xmake-io / xmake

Fix Clang module builds using outdated build flags #5425