CMakePresets with ninja are loading vcvars multiple times on preset change

jeremicmilan commented 8 months ago

Brief Issue Summary

When using ninja as a generator and when you change the configure preset it will automatically detect that ninja and cl are not in the path and it will load the appropriate vcvars, which is expected. However, after the change it will unload and reload every type of preset and for each preset type (configure, build, test, package, workflow). The expectation is for the vcvars (which is taking 5-10 seconds) to be ran only once per configure preset change, as the same Visual Studio development environment variables can be used for all presets. This behavior is significantly impacting the experience of doing builds with multiple compilers on Windows. For comparison on Linux, initializing all types of presets takes less than a second.

Debug Log

Relevant logs:

[proc] The command: C:\Windows\System32\where.exe cl exited with code: 1
[preset] Using developer environment from Visual Studio (instance b89de9c3, version 17.8.34601.278, installed at "C:\Program Files\Microsoft Visual Studio\2022\Enterprise")
[visual-studio] Patch Windows SDK path from C:\Program Files (x86)\Windows Kits\10\bin\x64 to C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64 for C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat
[proc] The command: C:\Windows\System32\where.exe ninja exited with code: 1
[preset] Ninja is not set on PATH, trying to use C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja\ninja.exe
[driver] Switching to configure preset: windows-ninja-msvc-x64
[driver] Unsetting test preset
[driver] Switching to build preset: windows-ninja-msvc-x64-debug
[proc] The command: C:\Windows\System32\where.exe cl exited with code: 1
[preset] Using developer environment from Visual Studio (instance b89de9c3, version 17.8.34601.278, installed at "C:\Program Files\Microsoft Visual Studio\2022\Enterprise")
[visual-studio] Patch Windows SDK path from C:\Program Files (x86)\Windows Kits\10\bin\x64 to C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64 for C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat
[proc] The command: C:\Windows\System32\where.exe ninja exited with code: 1
[preset] Ninja is not set on PATH, trying to use C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja\ninja.exe
[driver] Switching to test preset: __defaultTestPreset__
[proc] The command: C:\Windows\System32\where.exe cl exited with code: 1
[preset] Using developer environment from Visual Studio (instance b89de9c3, version 17.8.34601.278, installed at "C:\Program Files\Microsoft Visual Studio\2022\Enterprise")
[visual-studio] Patch Windows SDK path from C:\Program Files (x86)\Windows Kits\10\bin\x64 to C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64 for C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat
[proc] The command: C:\Windows\System32\where.exe ninja exited with code: 1
[preset] Ninja is not set on PATH, trying to use C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja\ninja.exe
[driver] Switching to package preset: __defaultPackagePreset__
[proc] The command: C:\Windows\System32\where.exe cl exited with code: 1
[preset] Using developer environment from Visual Studio (instance b89de9c3, version 17.8.34601.278, installed at "C:\Program Files\Microsoft Visual Studio\2022\Enterprise")
[visual-studio] Patch Windows SDK path from C:\Program Files (x86)\Windows Kits\10\bin\x64 to C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64 for C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat
[proc] The command: C:\Windows\System32\where.exe ninja exited with code: 1
[preset] Ninja is not set on PATH, trying to use C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja\ninja.exe
[driver] Switching to workflow preset: __defaultWorkflowPreset__

Additional Information

CMakePresets.json:

{
  "version": 8,
  "configurePresets": [
    {
      "name": "base",
      "hidden": true,
      "binaryDir": "${sourceDir}/out/build/${presetName}",
      "installDir": "${sourceDir}/out/install/${presetName}",
      "cacheVariables": {
        "CMAKE_DIR": "${fileDir}"
      }
    },
    {
      "name": "windows-x64",
      "hidden": true,
      "inherits": "base",
      "condition": {
        "type": "equals",
        "lhs": "${hostSystemName}",
        "rhs": "Windows"
      },
      "architecture": {
        "value": "x64",
        "strategy": "external"
      },
      "toolset": {
        "value": "v143,host=x64",
        "strategy": "external"
      }
    },
    {
      "name": "windows-ninja-x64",
      "hidden": true,
      "inherits": "windows-x64",
      "generator": "Ninja Multi-Config"
    },
    {
      "name": "windows-ninja-clang-x64",
      "displayName": "Windows Ninja Clang x64",
      "inherits": "windows-ninja-x64",
      "cacheVariables": {
        "CMAKE_C_COMPILER": "clang-cl.exe",
        "CMAKE_CXX_COMPILER": "clang-cl.exe"
      }
    },
    {
      "name": "windows-ninja-msvc-x64",
      "displayName": "Windows Ninja MSVC x64",
      "inherits": "windows-ninja-x64",
      "cacheVariables": {
        "CMAKE_C_COMPILER": "cl.exe",
        "CMAKE_CXX_COMPILER": "cl.exe"
      }
    },
    {
      "name": "windows-vs-msvc-x64",
      "displayName": "Windows VS MSVC x64",
      "description": "Generates Visual Studio solution and project files. Slower build than ninja.",
      "inherits": "windows-x64",
      "generator": "Visual Studio 17 2022"
    },
    {
      "name": "linux-ninja-x64",
      "hidden": true,
      "inherits": "base",
      "condition": {
        "type": "equals",
        "lhs": "${hostSystemName}",
        "rhs": "Linux"
      },
      "generator": "Ninja Multi-Config",
      "cacheVariables": {
        "CMAKE_C_FLAGS": "-m64",
        "CMAKE_CXX_FLAGS": "-m64"
      }
    },
    {
      "name": "linux-ninja-clang-x64",
      "displayName": "Linux Ninja Clang x64",
      "inherits": "linux-ninja-x64",
      "cacheVariables": {
        "CMAKE_C_COMPILER": "/usr/bin/clang",
        "CMAKE_CXX_COMPILER": "/usr/bin/clang++"
      }
    },
    {
      "name": "linux-ninja-gcc-x64",
      "displayName": "Linux Ninja GCC x64",
      "inherits": "linux-ninja-x64",
      "cacheVariables": {
        "CMAKE_C_COMPILER": "/usr/bin/gcc",
        "CMAKE_CXX_COMPILER": "/usr/bin/g++"
      }
    }
  ],
  "buildPresets": [
    {
      "name": "windows-ninja-clang-x64-debug",
      "displayName": "Windows Ninja Clang x64 Debug",
      "configurePreset": "windows-ninja-clang-x64",
      "configuration": "Debug"
    },
    {
      "name": "windows-ninja-clang-x64-release",
      "displayName": "Windows Ninja Clang x64 Release",
      "configurePreset": "windows-ninja-clang-x64",
      "configuration": "Release"
    },
    {
      "name": "windows-ninja-clang-x64-RelWithDebInfo",
      "displayName": "Windows Ninja Clang x64 RelWithDebInfo",
      "configurePreset": "windows-ninja-clang-x64",
      "configuration": "RelWithDebInfo"
    },
    {
      "name": "windows-ninja-msvc-x64-debug",
      "displayName": "Windows Ninja MSVC x64 Debug",
      "configurePreset": "windows-ninja-msvc-x64",
      "configuration": "Debug"
    },
    {
      "name": "windows-ninja-msvc-x64-release",
      "displayName": "Windows Ninja MSVC x64 Release",
      "configurePreset": "windows-ninja-msvc-x64",
      "configuration": "Release"
    },
    {
      "name": "windows-ninja-msvc-x64-RelWithDebInfo",
      "displayName": "Windows Ninja MSVC x64 RelWithDebInfo",
      "configurePreset": "windows-ninja-msvc-x64",
      "configuration": "RelWithDebInfo"
    },
    {
      "name": "windows-vs-msvc-x64-debug",
      "displayName": "Windows VS MSVC x64 Debug",
      "configurePreset": "windows-vs-msvc-x64",
      "configuration": "Debug"
    },
    {
      "name": "windows-vs-msvc-x64-release",
      "displayName": "Windows VS MSVC x64 Release",
      "configurePreset": "windows-vs-msvc-x64",
      "configuration": "Release"
    },
    {
      "name": "windows-vs-msvc-x64-RelWithDebInfo",
      "displayName": "Windows VS MSVC x64 RelWithDebInfo",
      "configurePreset": "windows-vs-msvc-x64",
      "configuration": "RelWithDebInfo"
    },
    {
      "name": "linux-ninja-clang-x64-debug",
      "displayName": "Linux Ninja Clang x64 Debug",
      "configurePreset": "linux-ninja-clang-x64",
      "configuration": "Debug"
    },
    {
      "name": "linux-ninja-clang-x64-release",
      "displayName": "Linux Ninja Clang x64 Release",
      "configurePreset": "linux-ninja-clang-x64",
      "configuration": "Release"
    },
    {
      "name": "linux-ninja-clang-x64-RelWithDebInfo",
      "displayName": "Linux Ninja Clang x64 RelWithDebInfo",
      "configurePreset": "linux-ninja-clang-x64",
      "configuration": "RelWithDebInfo"
    },
    {
      "name": "linux-ninja-gcc-x64-debug",
      "displayName": "Linux Ninja GCC x64 Debug",
      "configurePreset": "linux-ninja-gcc-x64",
      "configuration": "Debug"
    },
    {
      "name": "linux-ninja-gcc-x64-release",
      "displayName": "Linux Ninja GCC x64 Release",
      "configurePreset": "linux-ninja-gcc-x64",
      "configuration": "Release"
    },
    {
      "name": "linux-ninja-gcc-x64-RelWithDebInfo",
      "displayName": "Linux Ninja GCC x64 RelWithDebInfo",
      "configurePreset": "linux-ninja-gcc-x64",
      "configuration": "RelWithDebInfo"
    }
  ]
}

CMake Tools Diagnostics

{
  "os": "win32",
  "vscodeVersion": "1.86.2",
  "cmtVersion": "1.17.17",
  "configurations": [
    {
      "folder": "<scrubbed>",
      "cmakeVersion": "3.28.3",
      "configured": true,
      "generator": "Ninja Multi-Config",
      "usesPresets": true,
      "compilers": {
        "C": "cl.exe",
        "CXX": "cl.exe"
      }
    }
  ],
  "cpptoolsIntegration": {
    "isReady": true,
    "hasCodeModel": true,
    "activeBuildType": "Debug",
    "buildTypesSeen": [
      "Debug",
      "Release",
      "RelWithDebInfo"
    ],
    "requests": [],
    "responses": [],
    "partialMatches": [],
    "targetCount": 14,
    "executablesCount": 2,
    "librariesCount": 9,
    "targets": [<scrubbed>]
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": false
    }
  ]
}

gcampbell-msft commented 8 months ago

@jeremicmilan Thank you for notifying us of this. That is definitely not desirable to see this much output, and more importantly, to see a big slowdown when switching between presets and/or saving updates to presets.

We will investigate this, and based on workload and backlog, prioritize this so we don't have bad performance.

nmoreaud commented 8 months ago

I confirm that I have this behavior too, initialization is very slow.

But more globally, choosing an MS compiler version with CMake/Ninja and make it work on multiple IDEs is not an easy task. Ex :

If I call vcvars from a batch script and then run code, it works well... except if VSCode was already open on a WSL project. Then the environment variables seem to be shared between the 2 windows.
Ex: Edit User-Local CMake Kits command doesn't exist (probably because I use cmake presets?): https://github.com/microsoft/vscode-cmake-tools/blob/main/docs/kits.md#user-local-kits

We need to set redundant information, ex to say "use VS2022 toolset + VS2019 compiler + 64bits + Ninja MultiConfig":

    {
        "name": "windows",
        "cacheVariables": {
            "CMAKE_C_COMPILER": "cl.exe",
            "CMAKE_CXX_COMPILER": "cl.exe",
            "CMAKE_MAKE_PROGRAM": "ninja.exe"
        },
        "binaryDir": "${sourceDir}/build",
        "condition": {
            "type": "equals",
            "lhs": "${hostSystemName}",
            "rhs": "Windows"
        },
        "generator": "Ninja Multi-Config",
        "toolset": {
            "value": "v143,version=14.2,host=x64",
            "strategy": "external"
        },
        "architecture": {
            "value": "x64",
            "strategy": "external"
        }
    }

The documentation is very light on the subject, maybe a tutorial/use case example would help.

nmoreaud commented 8 months ago

Also, I wonder if vcvars result could be put in a cache file, because it is slow. I did this before finding about CMakePresets toolset support:

setenv.ps1:

if (!(Test-Path $PSScriptRoot\build\vcvars.txt)) {
    if (!(Test-Path $PSScriptRoot\build)) {
        mkdir $PSScriptRoot\build
    }
    # vs2022 compiles with 2019 tools + recent CMake
    cmd.exe /c "call `"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\Build\vcvars64.bat`" -vcvars_ver=14.2 && set > $PSScriptRoot\build\vcvars.txt"
}

Get-Content "$PSScriptRoot\build\vcvars.txt" | Foreach-Object {
    if ($_ -match "^(.*?)=(.*)$") {
        Set-Content "env:\$($matches[1])" $matches[2]
    }
}

gcampbell-msft commented 7 months ago

@nmoreaud Thanks for the follow up. I can confirm that we know about this issue and have a fix. We had been waiting for some other changes that we planned to initiate it, but to confirm that what I have locally fixes the issues, could you test with this vsix?

cmake-tools.zip

You'll need to download it, change the extension to .vsix, and then install it manually. Thanks!

jeremicmilan commented 7 months ago

@gcampbell-msft, I can confirm that the fix works for Ninja+MSVC. Thanks! It's awesome to see the switch working quickly, as well, as being cached when switching back and forth (vcvars not executed on the switch every time). When can we expect this to land into the public release of the extension?

Another problem I started hitting recently, not sure if related: Ninja+Clang is not loading the VS environment. I believe this worked a month or two ago, but it stopped working few weeks ago. It should not be a bug in the presets, as I set the environment to be used in windows-x64 base preset, which windows-ninja-clang-x64 inherits from (transitively).

[driver] Switching to configure preset: windows-ninja-clang-x64
[driver] Unsetting test preset
[driver] Switching to build preset: windows-ninja-clang-x64-debug
[driver] Switching to test preset: __defaultTestPreset__
[driver] Switching to package preset: __defaultPackagePreset__
[driver] Switching to workflow preset: __defaultWorkflowPreset__
[main] Configuring project: <scrubbed> 
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" -DCMAKE_C_COMPILER=clang-cl.exe -DCMAKE_CXX_COMPILER=clang-cl.exe -DCMAKE_DIR=<scrubbed>\cmake -DCMAKE_INSTALL_PREFIX=<scrubbed>/out/install/windows-ninja-clang-x64 -S<scrubbed> -B<scrubbed>/out/build/windows-ninja-clang-x64 -G "Ninja Multi-Config"
[cmake] CMake Error: CMake was unable to find a build program corresponding to "Ninja Multi-Config".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
[cmake] -- Configuring incomplete, errors occurred!
[proc] The command: "C:\Program Files\CMake\bin\cmake.EXE" -DCMAKE_C_COMPILER=clang-cl.exe -DCMAKE_CXX_COMPILER=clang-cl.exe -DCMAKE_DIR=<scrubbed>\cmake -DCMAKE_INSTALL_PREFIX=<scrubbed>/out/install/windows-ninja-clang-x64 -S<scrubbed> -B<scrubbed>/out/build/windows-ninja-clang-x64 -G "Ninja Multi-Config" exited with code: 1

Any chance this is related? Do you want me to open a new issue for this?

gcampbell-msft commented 7 months ago

@jeremicmilan As a rule of thumb, it's easier to track and investigate issues when they are created by themselves (assuming it's not a duplicate). So, if you could, please create a new issue.

If this is working for Ninja + MSVC but not for clang, I tend to agree this is likely a bug, so hopefully you can provide a minimal repro when you create the new issue. Thanks!

gcampbell-msft commented 7 months ago

@jeremicmilan As for when it might appear in the official release channel, we have a release schedule here: https://github.com/microsoft/vscode-cmake-tools/wiki/Release-schedule

jeremicmilan commented 7 months ago

@gcampbell-msft , the release version of the extension is now working way faster than before (2-3 seconds E2E compared to 10ish seconds from before). Previously there was a long delay after the following log entry:

[preset] Using developer environment from Visual Studio (instance 3d356f0c, version 17.9.34714.143, installed at "C:\Program Files\Microsoft Visual Studio\2022\Enterprise")

So, release version is more usable now, but it will be awesome to see your fix land as well, as then it's going to be super-fast!

Regarding the other issue, while I was trying to figure out the repro, I figured out that it happens in the pre-release version only (so I guess I started hitting it after I installed the version of the extension you shared). However, with your version, the issue happened only for Ninja+Clang, but in the pre-release version it happens for Ninja+MSVC as well. This is a serious issue, and I can't imagine it not being fixed until the release version. I guess pre-release version is very volatile, does it even make sense to file an issue for it?

gcampbell-msft commented 7 months ago

@jeremicmilan Yes, if the failure is happening in the pre-release, this is actually all the more reason to file the issue, because we can then fix the issue before releasing a bug in the official release. Additionally, it will help give me full context, right now I don't fully understand what the issue you hit that is happening with Ninja + Clang. Thanks

gcampbell-msft commented 7 months ago

@jeremicmilan I've quickly attempted to try to repro what you mentioned about the environment not being included for Ninja + MSVC, and I'm not sure I notice a difference between versions like 1.16.32 and the pre-release version, so any info you can provide so that we can investigate would be great. Thanks.

jeremicmilan commented 7 months ago

@gcampbell-msft, let me try and figure out minimal repro and create a new issue.

gcampbell-msft commented 7 months ago

@jeremicmilan Correction, I think I've found the issue. Could you test this vsix? You'll have to change the extension from .zip to .vsix and install it manually.

cmake-tools.zip

If you can confirm that this fixes your issue, then I can merge the PR I have open for it: #3669, and it can fix the pre-release channel for you.

jeremicmilan commented 7 months ago

Tested it briefly (about 10-15 times over different projects) and it works perfectly and very quickly! Thank you for the quick fix!

I couldn't figure out minimal repro, as it seems it was not deterministic. Your linked PR shed some light on that.

microsoft / vscode-cmake-tools