search

xmake-io / xmake

🔥 A cross-platform build utility based on Lua
https://xmake.io
Apache License 2.0
10.06k stars 785 forks source link

vsxmake project generator, very slow #5573

Open maximegmd opened 1 month ago

maximegmd commented 1 month ago

Xmake Version

2.9.4

Operating System Version and Architecture

Windows 11

Describe Bug

When running xmake project -k vsxmake on a project with 50 000 source file, it runs for hours.

This does not happen with the old vs generator. Running xmake directly to build is also not affected.

Expected Behavior

It doesn't take hours.

Project Configuration

n/a

Additional Information and Error Logs

n/a

waruqi commented 1 month ago

you can set -m mode and -a arch to reduce it.

maximegmd commented 1 month ago

you can set -m mode and -a arch to reduce it.

Tried it, canceled the job after 30 minutes, it didn't help.

waruqi commented 1 month ago

please provide verbose logs

maximegmd commented 1 month ago

There are no logs, it just hangs for hours after the initial checks:

(base) PS F:\Dev\HogwartsMods\Hogwarp> xmake project -k vsxmake -vD
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.41.34120
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checking for unzip ... ok
checking for git ... ok
checking for gzip ... ok
checking for tar ... ok
git rev-parse HEAD
finding mem from xmake ..
checking for xmake::mem ... mem 1.0.0
finding imgui from xmake ..
checking for xmake::imgui ... imgui v1.91.0
finding xbyak from xmake ..
checking for xmake::xbyak ... xbyak v7.06
finding minhook from xmake ..
checking for xmake::minhook ... minhook v1.3.3
checking for aqt ... no
checking for Qt SDK directory ... C:/Users/Work/AppData/Local/.xmake/packages/q/qt6base/6.8.0/b40ce8b80f0449248b45b9fbf1bcfe6e
checking for Qt SDK version ... 6.8.0
checking for xmake-repo::qt6base ... qt6base 6.8.0
checking for xmake-repo::qt6core ... qt6core 6.8.0
checking for xmake-repo::qt6gui ... qt6gui 6.8.0
checking for xmake-repo::qt6widgets ... qt6widgets 6.8.0
finding hopscotch-map from xmake ..
checking for xmake::hopscotch-map ... hopscotch-map v2.3.1
checking for cmake ... no
checking for cmake ... no
checking for cmake ... no
checking for cmake ... C:\Program Files\CMake\bin\cmake
finding snappy from xmake ..
checking for xmake::snappy ... snappy 1.1.10
finding mimalloc from xmake ..
checking for xmake::mimalloc ... mimalloc 2.1.7
finding openssl from xmake ..
checking for xmake::openssl ... openssl 1.1.1-w
finding abseil from xmake ..
checking for xmake::abseil ... abseil 20240722.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 28.0
finding gamenetworkingsockets from xmake ..
checking for xmake::gamenetworkingsockets ... gamenetworkingsockets v1.4.1
finding catch2 from xmake ..
checking for xmake::catch2 ... catch2 v2.13.9
finding libuv from xmake ..
checking for xmake::libuv ... libuv v1.48.0
finding spdlog from xmake ..
checking for xmake::spdlog ... spdlog v1.14.1
finding cryptopp from xmake ..
checking for xmake::cryptopp ... cryptopp 8.9.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 3.19.4
finding glm from xmake ..
checking for xmake::glm ... glm 1.0.1
finding zlib from xmake ..
checking for xmake::zlib ... zlib v1.3.1
finding nlohmann_json from xmake ..
checking for xmake::nlohmann_json ... nlohmann_json v3.11.3
finding dotnet from xmake ..
checking for xmake::dotnet ... dotnet 8.0.6
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for the c++ compiler (cxx) ... cl.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
generating code\encoding\BuildInfo.h.in to build\BuildInfo.h ..
  > replace GIT_BRANCH -> dev
  > replace GIT_COMMIT -> ec9964cf
generating code\encoding\BuildInfo.h.in ... cache
configure
{
    ndk_stdcxx = true
    mode = release
    kind = static
    plat = windows
    host = windows
    qt = C:/Users/Work/AppData/Local/.xmake/packages/q/qt6base/6.8.0/b40ce8b80f0449248b45b9fbf1bcfe6e
    vs = 2022
    arch = x64
    buildir = build
    qt_sdkver = 6.8.0
    ccache = true
}
using project kind vs2022
checking for debug.x64 ...
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.41.34120
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checking for unzip ... ok
checking for git ... ok
checking for gzip ... ok
checking for tar ... ok
finding mem from xmake ..
checking for xmake::mem ... mem 1.0.0
finding imgui from xmake ..
checking for xmake::imgui ... imgui v1.91.0
finding xbyak from xmake ..
checking for xmake::xbyak ... xbyak v7.06
finding minhook from xmake ..
checking for xmake::minhook ... minhook v1.3.3
checking for aqt ... no
checking for Qt SDK directory ... C:/Users/Work/AppData/Local/.xmake/packages/q/qt6base/6.8.0/b40ce8b80f0449248b45b9fbf1bcfe6e
checking for Qt SDK version ... 6.8.0
checking for xmake-repo::qt6base ... qt6base 6.8.0
checking for xmake-repo::qt6core ... qt6core 6.8.0
checking for xmake-repo::qt6gui ... qt6gui 6.8.0
checking for xmake-repo::qt6widgets ... qt6widgets 6.8.0
finding hopscotch-map from xmake ..
checking for xmake::hopscotch-map ... hopscotch-map v2.3.1
checking for cmake ... no
checking for cmake ... no
checking for cmake ... no
checking for cmake ... C:\Program Files\CMake\bin\cmake
finding snappy from xmake ..
checking for xmake::snappy ... snappy 1.1.10
finding mimalloc from xmake ..
checking for xmake::mimalloc ... mimalloc 2.1.7
finding openssl from xmake ..
checking for xmake::openssl ... openssl 1.1.1-w
finding abseil from xmake ..
checking for xmake::abseil ... abseil 20240722.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 28.0
finding gamenetworkingsockets from xmake ..
checking for xmake::gamenetworkingsockets ... gamenetworkingsockets v1.4.1
finding catch2 from xmake ..
checking for xmake::catch2 ... catch2 v2.13.9
finding libuv from xmake ..
checking for xmake::libuv ... libuv v1.48.0
finding spdlog from xmake ..
checking for xmake::spdlog ... spdlog v1.14.1
finding cryptopp from xmake ..
checking for xmake::cryptopp ... cryptopp 8.9.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 3.19.4
finding glm from xmake ..
checking for xmake::glm ... glm 1.0.1
finding zlib from xmake ..
checking for xmake::zlib ... zlib v1.3.1
finding nlohmann_json from xmake ..
checking for xmake::nlohmann_json ... nlohmann_json v3.11.3
finding dotnet from xmake ..
checking for xmake::dotnet ... dotnet 8.0.6
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for the c++ compiler (cxx) ... cl.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for the c compiler (cc) ... cl.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
checking for release.x64 ...
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.41.34120
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checking for unzip ... ok
checking for git ... ok
checking for gzip ... ok
checking for tar ... ok
finding mem from xmake ..
checking for xmake::mem ... mem 1.0.0
finding imgui from xmake ..
checking for xmake::imgui ... imgui v1.91.0
finding xbyak from xmake ..
checking for xmake::xbyak ... xbyak v7.06
finding minhook from xmake ..
checking for xmake::minhook ... minhook v1.3.3
checking for aqt ... no
checking for Qt SDK directory ... C:/Users/Work/AppData/Local/.xmake/packages/q/qt6base/6.8.0/b40ce8b80f0449248b45b9fbf1bcfe6e
checking for Qt SDK version ... 6.8.0
checking for xmake-repo::qt6base ... qt6base 6.8.0
checking for xmake-repo::qt6core ... qt6core 6.8.0
checking for xmake-repo::qt6gui ... qt6gui 6.8.0
checking for xmake-repo::qt6widgets ... qt6widgets 6.8.0
finding hopscotch-map from xmake ..
checking for xmake::hopscotch-map ... hopscotch-map v2.3.1
checking for cmake ... no
checking for cmake ... no
checking for cmake ... no
checking for cmake ... C:\Program Files\CMake\bin\cmake
finding snappy from xmake ..
checking for xmake::snappy ... snappy 1.1.10
finding mimalloc from xmake ..
checking for xmake::mimalloc ... mimalloc 2.1.7
finding openssl from xmake ..
checking for xmake::openssl ... openssl 1.1.1-w
finding abseil from xmake ..
checking for xmake::abseil ... abseil 20240722.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 28.0
finding gamenetworkingsockets from xmake ..
checking for xmake::gamenetworkingsockets ... gamenetworkingsockets v1.4.1
finding catch2 from xmake ..
checking for xmake::catch2 ... catch2 v2.13.9
finding libuv from xmake ..
checking for xmake::libuv ... libuv v1.48.0
finding spdlog from xmake ..
checking for xmake::spdlog ... spdlog v1.14.1
finding cryptopp from xmake ..
checking for xmake::cryptopp ... cryptopp 8.9.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 3.19.4
finding glm from xmake ..
checking for xmake::glm ... glm 1.0.1
finding zlib from xmake ..
checking for xmake::zlib ... zlib v1.3.1
finding nlohmann_json from xmake ..
checking for xmake::nlohmann_json ... nlohmann_json v3.11.3
finding dotnet from xmake ..
checking for xmake::dotnet ... dotnet 8.0.6
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for the c++ compiler (cxx) ... cl.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for the c compiler (cc) ... cl.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
checking for releasedbg.x64 ...
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.41.34120
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(zig.exe version), No such file or directory
checking for zig ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checkinfo: cannot runv(nim.exe --version), No such file or directory
checking for nim ... no
checking for unzip ... ok
checking for git ... ok
checking for gzip ... ok
checking for tar ... ok
finding mem from xmake ..
checking for xmake::mem ... mem 1.0.0
finding imgui from xmake ..
checking for xmake::imgui ... imgui v1.91.0
finding xbyak from xmake ..
checking for xmake::xbyak ... xbyak v7.06
finding minhook from xmake ..
checking for xmake::minhook ... minhook v1.3.3
checking for aqt ... no
checking for Qt SDK directory ... C:/Users/Work/AppData/Local/.xmake/packages/q/qt6base/6.8.0/b40ce8b80f0449248b45b9fbf1bcfe6e
checking for Qt SDK version ... 6.8.0
checking for xmake-repo::qt6base ... qt6base 6.8.0
checking for xmake-repo::qt6core ... qt6core 6.8.0
checking for xmake-repo::qt6gui ... qt6gui 6.8.0
checking for xmake-repo::qt6widgets ... qt6widgets 6.8.0
finding hopscotch-map from xmake ..
checking for xmake::hopscotch-map ... hopscotch-map v2.3.1
checking for cmake ... no
checking for cmake ... no
checking for cmake ... no
checking for cmake ... C:\Program Files\CMake\bin\cmake
finding snappy from xmake ..
checking for xmake::snappy ... snappy 1.1.10
finding mimalloc from xmake ..
checking for xmake::mimalloc ... mimalloc 2.1.7
finding openssl from xmake ..
checking for xmake::openssl ... openssl 1.1.1-w
finding abseil from xmake ..
checking for xmake::abseil ... abseil 20240722.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 28.0
finding gamenetworkingsockets from xmake ..
checking for xmake::gamenetworkingsockets ... gamenetworkingsockets v1.4.1
finding catch2 from xmake ..
checking for xmake::catch2 ... catch2 v2.13.9
finding libuv from xmake ..
checking for xmake::libuv ... libuv v1.48.0
finding spdlog from xmake ..
checking for xmake::spdlog ... spdlog v1.14.1
finding cryptopp from xmake ..
checking for xmake::cryptopp ... cryptopp 8.9.0
finding protobuf-cpp from xmake ..
checking for xmake::protobuf-cpp ... protobuf-cpp 3.19.4
finding glm from xmake ..
checking for xmake::glm ... glm 1.0.1
finding zlib from xmake ..
checking for xmake::zlib ... zlib v1.3.1
finding nlohmann_json from xmake ..
checking for xmake::nlohmann_json ... nlohmann_json v3.11.3
finding dotnet from xmake ..
checking for xmake::dotnet ... dotnet 8.0.6
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for the c++ compiler (cxx) ... cl.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
checking for cl.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\cl.exe
checking for the c compiler (cc) ... cl.exe
checking for link.exe ... C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\bin\HostX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
waruqi commented 1 month ago

you can try this way to get backtrace. https://github.com/xmake-io/xmake/issues/2033

maximegmd commented 1 month ago

Did a few runs and here are the traces

[xmake]: [engine]: stack traceback:
        [C]: in function 'type'
        @programdir\core\base\table.lua:98: in function 'table.join'
        @programdir\plugins\project\vsxmake\render.lua:76: in global '_expand'
        @programdir\plugins\project\vsxmake\render.lua:89: in function <@programdir\plugins\project\vsxmake\render.lua:87>
        (...tail calls...)
        [C]: in function 'string.gsub'
        @programdir\plugins\project\vsxmake\render.lua:93: in function <@programdir\plugins\project\vsxmake\render.lua:87>
        (...tail calls...)
        @programdir\plugins\project\vsxmake\vsxmake.lua:247: in field '?'
        @programdir\plugins\project\main.lua:75: in global '_make'
        @programdir\plugins\project\main.lua:95: in function <@programdir\plugins\project\main.lua:78>
        (...tail calls...)
        [C]: in function 'xpcall'
        @programdir\core\base\utils.lua:275: in function 'sandbox/modules/utils.trycall'
        (...tail calls...)
        @programdir\core\base\task.lua:491: in function 'base/task.run'
        @programdir\core\main.lua:327: in upvalue 'cotask'
        @programdir\core\base\scheduler.lua:406: in function <@programdir\core\base\scheduler.lua:399>
[xmake]: [engine]: stack traceback:
        @programdir\core\base\table.lua:98: in function 'sandbox/modules/table.join'
        @programdir\core\base\table.lua:97: in function 'sandbox/modules/table.join'
        @programdir\plugins\project\vsxmake\render.lua:76: in global '_expand'
        @programdir\plugins\project\vsxmake\render.lua:89: in function <@programdir\plugins\project\vsxmake\render.lua:87>
        (...tail calls...)
        [C]: in function 'string.gsub'
        @programdir\plugins\project\vsxmake\render.lua:93: in function <@programdir\plugins\project\vsxmake\render.lua:87>
        (...tail calls...)
        @programdir\plugins\project\main.lua:75: in global '_make'
        @programdir\plugins\project\main.lua:95: in function <@programdir\plugins\project\main.lua:78>
        (...tail calls...)
        @programdir\plugins\project\main.lua:95: in function <@programdir\plugins\project\main.lua:78>
        (...tail calls...)
        [C]: in function 'xpcall'
        @programdir\core\base\utils.lua:275: in function 'sandbox/modules/utils.trycall'
        (...tail calls...)
        @programdir\core\base\task.lua:491: in function 'base/task.run'
        @programdir\core\base\scheduler.lua:406: in function <@programdir\core\base\scheduler.lua:399>
        @programdir\core\base\scheduler.lua:406: in function <@programdir\core\base\scheduler.lua:399>
[xmake]: [engine]: stack traceback:
        [C]: in function 'type'
        @programdir\core\base\table.lua:98: in function 'base/table.join'
        @programdir\plugins\project\vsxmake\render.lua:76: in global '_expand'
        [C]: in function 'base/string.gsub'
        [C]: in function 'base/string.gsub'
        @programdir\plugins\project\vsxmake\render.lua:93: in function <@programdir\plugins\project\vsxmake\render.lua:87>
        (...tail calls...)
        @programdir\plugins\project\vsxmake\vsxmake.lua:247: in field '?'
        @programdir\plugins\project\main.lua:95: in function <@programdir\plugins\project\main.lua:78>
        (...tail calls...)
        @programdir\plugins\project\main.lua:95: in function <@programdir\plugins\project\main.lua:78>
        (...tail calls...)
        [C]: in function 'xpcall'
        @programdir\core\base\utils.lua:275: in function 'base/utils.trycall'
        (...tail calls...)
        @programdir\core\base\task.lua:491: in function 'base/task.run'
        @programdir\core\base\scheduler.lua:406: in function <@programdir\core\base\scheduler.lua:399>
        @programdir\core\base\scheduler.lua:406: in function <@programdir\core\base\scheduler.lua:399>
waruqi commented 1 month ago

Maybe we should optimize here, big projects can cause it to be very slow.

I know this place can continue to be optimized, but at the moment I can't optimize it very well.

This patch (https://github.com/xmake-io/xmake/pull/5588) I made some attempts to optimize, you can try it as well and see if it improves.

You can also try to debug it or help optimize it.

https://github.com/xmake-io/xmake/blob/7ae40ebaaba975239e4550159fd440bc94237764/xmake/plugins/project/vsxmake/render.lua#L62-L96

@OpportunityLiu @SirLynix Any idea?

SirLynix commented 1 month ago

I think we should remake the vsxmake generator by forking vs generator and putting idea I had in another issue (use nmake settings to trigger xmake, like Unreal Engine does)

waruqi commented 1 month ago

I think we should remake the vsxmake generator by forking vs generator and putting idea I had in another issue (use nmake settings to trigger xmake, like Unreal Engine does)

You can try it, but for now I can only try to optimize it first.

@maximegmd Will the new patch be faster?

maximegmd commented 1 month ago

@waruqi I am unable to run as I am hitting this https://github.com/xmake-io/xmake-repo/issues/5180 after a clean install

waruqi commented 1 month ago

@waruqi I am unable to run as I am hitting this xmake-io/xmake-repo#5180 after a clean install

try it again.

https://github.com/xmake-io/xmake-repo/issues/5180#issuecomment-2339494658

maximegmd commented 1 month ago

This helps a lot; it now takes 10 minutes vs a few hours before.

That being said, I think we should keep this issue open to track vsxmake improvements, even 10 minutes is quite slow.

SirLynix commented 1 month ago

how long does vs generator take?

maximegmd commented 1 month ago

Less than a minute

waruqi commented 1 month ago

Can you provide a new stuck backtrace?