search

devcontainers / cli

A reference implementation for the specification that can create and configure a dev container from a devcontainer.json.
https://containers.dev
MIT License
1.53k stars 217 forks source link

Ability to run a specific global scenario test #227

Open virtualstaticvoid opened 2 years ago

virtualstaticvoid commented 2 years ago

Hello,

Following on from #163 I converted all my features which use installsAfter to use the global scenarios test functionality and removed them from specific feature tests since they always fail for the default test.

Whilst in my example for #163 I only had one feature, my work project has over 20 features.

The issue now is that the GitHub Actions build times out (>6 hours) running the global scenarios for 20+ features.

This issue could be solved in one of the following ways:

  1. Allow running a specific global scenario test, providing it's name on the command line, such as with a command line switch --global-scenario=<name-of-scenario>. This would enable use of a build matrix for global scenarios too, such as the one used in feature-template to run specific feature tests individually. This would also aid debugging failed scenarios and allow them to be run in parallel.

  2. Use a scenarios.json within the features's test directory. If the file is present, then the "default" feature test should not run, since it won't pass anyway, and the included scenarios be run instead. Global scenarios wouldn't be needed in this case.

I would think option 2 would be the preferred approach, as it shouldn't break any existing implementations and as I understand global scenarios are intended for testing whether a combination of features work together instead of for features which simply use installsAfter. Having a scenarios.json file within a features test directory seems more obvious and natural place.

AlexanderLanin commented 2 years ago

It's kind of the same for testing a single feature with a "big" scenarios.json. It would be nice to run those in parallel. I would like to / I do run this with every line changed of the feature, but it easily takes 20+ minutes to go through all the scenarios from scenarios.json.

I have another solution proposal: run all the scenarios tests in parallel, and if/once a test fails, print its output (and abort the others).

NicoVIII commented 1 year ago

As far as I can see this (2.) is possible now, although in my experience you have to provide --skip-autogenerated, if you only have scenarios. https://github.com/devcontainers/cli/pull/124