Parallelization within executable

[helloworldhello762976h9f7tb34y8t7]

Is your feature request related to a problem? Please describe. Running Google Test or Catch2 only runs executeables in parallel, which necessitates splitting test files into one file for each test.

Describe the solution you'd like Catch2 parallelization is infeasible but google test has gtest-parallel, ideally hitting the play button should run tests in parallel the same way executeables are but using the gtest-parallel script

Additional context This is more a question of whether you think it's feasible. I don't know Typescript and haven't developed extensions before but would be willing to put in the work if it's feasible. Otherwise I don't want to waste the time. And if it's simple enough maybe you wouldn't mind?

[helloworldhello762976h9f7tb34y8t7]

Thought it was a duplicate, but may still be worth considering

[matepek]

Catch2 parallelization is infeasible

Why do you say that? I think Catch2 parallelization is the same as gtest. Both of them are feasible. But it is a big structural change.

And giving a proper solution is also not trivial. For example how would you split the tests in an executable? Natural ordering seems the first, most convenient choice but I'm pretty sure somebody will come and say that he went to do it differently. Note that starting an executable is usually more resource intensive than running one test case.

Otherwise it is a great idea. I think my users would appreciate a feature like this.

Wait a sec... I have an alternative solution ... let me think about it..

[helloworldhello762976h9f7tb34y8t7]

Well, I say infeasible and not impossible because I thought it would probably be too much work, but I don't know for sure. With GTest it seems you should be able to fire off gtest-parallel pretty easily.

In terms of splitting the tests, at least with Catch2 you could just specify that the users can add tags to their tests for grouping else they'll fall into the default group which is natural ordering. With GTest you could maybe group by fixture? But I can easily see users wanting more granularity and not wanting to have to make more fixtures

[matepek]

Alternative solution: Allowing having the same executable more than once with some regex like:

"catch2TestExplorer.executables": [
    {
                "name": "${filename} / group 1",
        "pattern": "./**/{t,T}est",
                "test_filter_regex": [".*drive.*"]
    },
    {
                "name": "${filename} / group 2",
        "pattern": "./**/{t,T}est",
                "test_filter_regex": [".*matrix.*"]
    }
]

This would give a similar result, easy to implement but I think it is error prone. For example one should make sure that every test case is parsed exactly once.

I like the idea. I think I will develop in a couple of months ("Thanks" to 🦠😬)

If you want to motivate me for shorter deadline you can

• show your open source contribution
• buy me coffees
• show any kind of your contribution which makes/made the world to a better place to live in. (Collecting garbage selectively counts. I wish that could be basic.)

Oh you've just commented. I will react to that in a separate comment.

[matepek]

If gtest have a script you can use that now. you just have to set up that the executable pattern finds that script and it will run it. I guess it is not trivial but definitely solvable already. But there are other problems with that. (See wrapper related closed issues.)

Still the first solution seems the best which spawns the same executable for multiple times. Pros:

• No drawbacks and limitations.
• Nicely customizable
• Nice business logic (I hope)
• All the frameworks would benefit from it.

[matepek]

I would like to mention that this feature only solves a problem caused by a common practice: putting "too much" test cases into one executable.

This pattern has pros and cons in general and in perspective of this extension too.

Anyhow the practice is widespread therefore I think this feature is justified.

[helloworldhello762976h9f7tb34y8t7]

Well, even if you have a bunch of files < 100 lines multiple workers would still give significant speedup, but also there's the fact that some people are dealing with tests other people wrote and can't spend the time to separate them all

[matepek]

Yep Yep,

Also sometime the test environment initialization can take too much time and it doesn't worth the separation because then the environment initialization time would be redundant and too much.

[matepek]

depends on:

• https://github.com/catchorg/Catch2/issues/1905
• https://github.com/catchorg/Catch2/issues/1912

[matepek]

Having some thought about the matter I got interested that how would you group the tests in an executable? I had some idea but I'm interested what would be useful for you especially.

Note: Some tests may requires exclusive access the same resources like writing the specific file, opening same port, etc.. One has to be careful with this so it definitely wont be the default option.

Did you see the new grouping feature? How about running groups together? We have to consider the startup overhead..

[matepek]

Something basic should work. Check parallelizationLimit