search

cta-wave / dpctf-test-runner

Repo for the DPCTF Test Runner
Other
0 stars 3 forks source link

make it easier to run a custom subset of tests #47

Open jpiesing opened 3 months ago

jpiesing commented 3 months ago

Running a custom subset of tests has been reported to be complex (e.g. the ones included in the HbbTV test suite).

FritzHeiden commented 3 months ago

A quick solution for this that comes to my mind is to extend the configuration page that allows to enter token (or part of it) of a previous session to load the same configuration into the form. Then you could run the session right away or make some adjustments

jpiesing commented 2 months ago

A quick solution for this that comes to my mind is to extend the configuration page that allows to enter token (or part of it) of a previous session to load the same configuration into the form. Then you could run the session right away or make some adjustments

I don't see this as addressing the proposal. Would it be possible to create a directory which can be (pre-)populated with files containing lists of test cases which are included in the test runner menu? The tests included in the HbbTV test suite could be one of these.

louaybassbouss commented 2 months ago

A quick solution for this that comes to my mind is to extend the configuration page that allows to enter token (or part of it) of a previous session to load the same configuration into the form. Then you could run the session right away or make some adjustments

I don't see this as addressing the proposal. Would it be possible to create a directory which can be (pre-)populated with files containing lists of test cases which are included in the test runner menu? The tests included in the HbbTV test suite could be one of these.

@jpiesing, a directory is what we used in dedicated branches (e.g., HbbTV) to facilitate test selection, particularly during events like the HbbTV Plugfest. However, the issue with the directory is that it mismatches the semantic grouping of tests in the UI, leading to potential duplications of tests because they belong to different groups. An idea could be to introduce a filter menu with the following items (just thinking aloud):

The important thing is not to add any versioning into the filter.

jpiesing commented 2 months ago

@jpiesing, a directory is what we used in dedicated branches (e.g., HbbTV) to facilitate test selection, particularly during events like the HbbTV Plugfest. However, the issue with the directory is that it mismatches the semantic grouping of tests in the UI, leading to potential duplications of tests because they belong to different groups. An idea could be to introduce a filter menu with the following items (just thinking aloud):

  • Subset: Options include All, HbbTV, ...
  • Review: Options include All, Validated, Beta, ...
  • Content: Options include All, Video, Audio, Video/Audio, ...
  • Duration: Options include All, Long Duration, Non-Long Duration, ...

The important thing is not to add any versioning into the filter.

In principle this sounds like a good idea. One drawback that occurs to me is that the test runner would need to know which of these applied to which test.

louaybassbouss commented 2 months ago

@jpiesing, a directory is what we used in dedicated branches (e.g., HbbTV) to facilitate test selection, particularly during events like the HbbTV Plugfest. However, the issue with the directory is that it mismatches the semantic grouping of tests in the UI, leading to potential duplications of tests because they belong to different groups. An idea could be to introduce a filter menu with the following items (just thinking aloud):

  • Subset: Options include All, HbbTV, ...
  • Review: Options include All, Validated, Beta, ...
  • Content: Options include All, Video, Audio, Video/Audio, ...
  • Duration: Options include All, Long Duration, Non-Long Duration, ...

The important thing is not to add any versioning into the filter.

In principle this sounds like a good idea. One drawback that occurs to me is that the test runner would need to know which of these applied to which test.

For Options Review, Content, Duration we can do it without additional information (we know which tests are validated or not, which are audio/video/both, long duration or not). For Subset filter with options like HbbTV yes we need to tell the test runner which tests we need. For this we need to introduce a subset filter configuration and it will be applied. I prefer to not hardcode anything related to HbbTV (or other organisations in the future) and keep the test suite as generic as possible.

jpiesing commented 2 months ago

@jpiesing, a directory is what we used in dedicated branches (e.g., HbbTV) to facilitate test selection, particularly during events like the HbbTV Plugfest. However, the issue with the directory is that it mismatches the semantic grouping of tests in the UI, leading to potential duplications of tests because they belong to different groups. An idea could be to introduce a filter menu with the following items (just thinking aloud):

  • Subset: Options include All, HbbTV, ...
  • Review: Options include All, Validated, Beta, ...
  • Content: Options include All, Video, Audio, Video/Audio, ...
  • Duration: Options include All, Long Duration, Non-Long Duration, ...

The important thing is not to add any versioning into the filter.

In principle this sounds like a good idea. One drawback that occurs to me is that the test runner would need to know which of these applied to which test.

For Options Review, Content, Duration we can do it without additional information (we know which tests are validated or not, which are audio/video/both, long duration or not). For Subset filter with options like HbbTV yes we need to tell the test runner which tests we need. For this we need to introduce a subset filter configuration and it will be applied. I prefer to not hardcode anything related to HbbTV (or other organisations in the future) and keep the test suite as generic as possible.

How big a divergence from the WPT test runner would this be? How much of this might be acceptable back into the WPT trunk? To what extent would this be useful to WMAS?

Lis-TV commented 2 months ago

The aim is to make a test material easy to find , deploy and use. First, HbbTV have to add manuals and instructions to a) tech.cta.wave.dpctf* XML, b) HbbTV site .

The manuals would include link to installation (dpctf-deploy) and list of materials (HbbTV-only). Selecting may be done in two ways: 1) prepare the list of materials to allow manual selection. 2) saved preselection/project/list/branch whatever - but must be easy to use (Louay?).

FritzHeiden commented 2 months ago

image

With https://github.com/cta-wave/dpctf-test-runner/pull/52 and https://github.com/cta-wave/dpctf-tests/pull/177 there is now a subset selection mechanism which features the hbbtv subset with local und online content. More subsets can be added quickly by adding them to the test-subsets.json