Discussion: CTest integration or external API?

[MathiasMagnus]

I saw that the roadmap has CTest integration on it and I was wondering what would be the best route to take in terms of UX. CMake Tools had this capability built-in, but I was wondering if it would be better to wire it in to a "3rd" partry extension, namely Test Explorer UI.

It has a dedicated UI for presenting tests from various frameworks, which is nice because it would feel familiar whether I'm using CTest, or another supported test framework (the list is LONG), plus it would declutter CMake Integrations own UI. It shouldn't be that hard to teach it another framework, as it has a stub project for teaching it another back-end. Also, mix'n'match of test frameworks in the same project would be a breeze, especially in multi-root contexts.

I was even thinking of making it my summer toy project to try to teach it CTest (I've never written a single line of TypeScript in my life, not to mention VS Code extension, so the success probability is extremely low). Should someone want to beat me to it, please comment here.

What do you think? Would you like if CTest had it's own dedicated UI via another extension?

[go2sh]

Hi there, sorry I was away from keyboard for some time.

Generally, I would step away from adding a new UI element to vscode. It is quiet painful and I'am not able to maintain it. :-/

I had a quick look at the text explorer UI extension and it supports catch2 already which is sufficient for me. But it could be a good possibility to add an ctest <-> test explorer ui adapter to this extensions. I'll have a look.

[MathiasMagnus]

I already started digging into how to obtain CTest info in a reliable and easy to parse manner. I have found that the cmake-file-api currently does not support obtaining test information. However, CTest can be invoked to dump a similar (it seems to be the exact kind what the file API provides) using ctest.exe --show-only=json-v1 with an optional -C <Config> flag to obtain configuration specific information.

A simple CMakeLists.txt file such as:

cmake_minimum_required(VERSION 3.8) # CXX_STANDARD 17

project(TestCTest LANGUAGES CXX)

add_executable(${PROJECT_NAME} Main.cpp)

set(BUILT_TESTING ON)
include(CTest)

add_test(NAME DefaultTest
         COMMAND ${PROJECT_NAME})

with -C Release results in the output:

{
  "backtraceGraph" : 
  {
    "commands" :
    [
      "add_test"
    ],
    "files" :
    [
      "C:/Users/mnagy/Source/TestCTest/CMakeLists.txt"
    ],
    "nodes" : 
    [
      {
        "file" : 0
      },
      {
        "command" : 0,
        "file" : 0,
        "line" : 10,
        "parent" : 0
      }
    ]
  },
  "kind" : "ctestInfo",
  "tests" :
  [
    {
      "backtrace" : 1,
      "command" :
      [
        "C:/Users/mnagy/Source/TestCTest/.vscode/build/TestCTest.exe"
      ],
      "config" : "Release",
      "name" : "DefaultTest",
      "properties" :
      [
        {
          "name" : "WORKING_DIRECTORY",
          "value" : "C:/Users/mnagy/Source/TestCTest/.vscode/build"
        }
      ]
    }
  ],
  "version" :
  {
    "major" : 1,
    "minor" : 0
  }
}

I believe this is enough to invoke CTest. Until the file API driver doesn't learn the ctestinfo kind, it requires some plumbing to ensure that the tests UI is never out of date. Ideally the adapter would drop in a query file and monitor the response file for changes which occurs whenever the user reconfigures the project. In the meantime I don't quite know when to invoke CTest. The adapter would have to listen to commands of other extensions, or watch CMakeCache.txt for changes, but that would require knowing where it is, which is yet again an option of other extensions.

What are your thoughts?