Test explorer sometimes doesn't find all tests

[edkelly303]

Given this test file:

module MyTest exposing (..)

import Expect
import Test exposing (..)

add1 : Int -> Int
add1 n =
    n + 1

suite : Test
suite =
    describe "Tests are here"
        [ describe "This one gets found"
            [ test "add1 1" <|
                \_ ->
                    add1 1
                        |> Expect.equal 2
            ]
        , describe "This one doesn't" <|
            List.map
                (\( label, input, output ) ->
                    test label <|
                        \_ ->
                            add1 input
                                |> Expect.equal output
                )
                [ ( "add1 2", 2, 3 )
                , ( "add1 3", 3, 4 )
                ]
        ]

Expected Behavior

I would expect the test explorer panel to find and display all three tests in the tests list when you open or refresh the panel.

Current Behavior

The explorer panel only finds the first test

If I click the button to run all tests, the explorer does then find and display all the tests, but as soon as I make any change to the test file, they disappear again.

Possible Solution

Not a solution, but I'm guessing this happens because the language server initially only recognizes literal tests when it scans the test files - not functions that will ultimately evaluate to be tests. So it might not be possible to detect where all the tests are without actually running the tests at least once?

Steps to Reproduce (for bugs)

See code above.

Context

I was just trying to avoid some boilerplate in my test functions - the real use case was tests for a parser, where I basically just need to compare a bunch of string inputs with a bunch of outputs. So the body of all the tests is identical, only the inputs and outputs change. So writing out all the test bodies by hand seemed like unnecessary work.

The current user experience is a bit suboptimal because if, for example, I make a change to one test and want to rerun that test only, I can't find and click it in the test explorer, because changing the code will have caused it to disappear. So instead I need to run all tests, and then expand all the nodes in the test explorer tree until I get back to the one test I wanted to look at.

Your Environment

• Version used: 2.4.1
• Editor name and version: VS Code 1.68.0
• Environment name and version: nodejs 16.13.0
• Operating System and version: Linux x64 5.15.0-37-generic (Xubuntu)

[razzeee]

Yes, we're not able to infer what the total test set is, without running the test suite. Walking the code only let's us see non generated tests.

[edkelly303]

I'm not sure how the code-walking works, and this could be a stupid suggestion... but could the walker add a node to the test explorer tree each time it finds a "describe", even if it can't actually find a test in the list underneath that describe?

[razzeee]

Yes, but I'm not sure if that's enough to start that specific test, would have to check when implementing.