propose how to identify beta / non-validated tests in the test runner UI

[jpiesing]

From the November 7th meeting, @louaybassbouss @fritz to make a proposal for how to label beta / non-validated tests in the test runner UI - something that is realistic and reflects how the test runner works. If the test operator selects beta tests, can they be given a warning?

[yanj-github]

One thing to consider is how Section 9 tests are labelled, as the video observations are validated, but not yet the audio ones. We could add a warning message on the audio observation results stating they are not validated/beta/pending? We just need to be consistent in the terminology we use in TR/OF/Readmes etc

[FritzHeiden]

For the labelling of the non-validated test I think the simplest way is to add some string to the test group name, like -non-validated, e.g. cfhd-sets_15_30_60-non-validated. This allows to easily group non-validated tests using the csv file for test generation. It will also be visible in test reports.

Based on the label in the test group name, the UI can detect selection of these tests. The way to show a warning would be similiar to this one:

The color of this banner will be a typical warning yellow with a text pointing out that the user selected non-validated tests. We may also include a link to an MD file in the github repos to allow for a more elaborate explanation.

What do you think?

Further points for discussion:

1. What label to use for test group names?
2. What information should be included in the warning text?
3. If we include a link, where should we put the document?

[gitwjr]

Close when the warnings are implemented for tests that have not been validated on multiple devices such as in a plugfest.

[gitwjr]

Labels for tests will be "Released" vs "Beta" with additional explanatory text linked. Need to provide what the message will be and when it will be displayed/texted/link provided. Email discussion.

[FritzHeiden]

I opened a PR with first implementation for this: https://github.com/cta-wave/dpctf-test-runner/pull/37

Message text still is placeholder

[gitwjr]

2023/12/19: Placeholders have been created for the Released and Beta test labels. If there is a Beta test included in a group the labels for each test will be displayed. The Beta label will include a shortcut link (noting "click here for more information" or similar) to additional descriptive text in the ReadMe which will open in a new tab. Similar links will be included for Warnings such as when running tests requiring HTTPS (e.g. encrypted content) using IP addresses instead of valid domains, etc.

[jpiesing]

@gitwjr @cta-source We have a proposal above with an implementation that is not yet integrated. Who needs to approve this? From my point of view, this is good enough.

[gitwjr]

Looks good to me. It meets what was proposed to TWG so I think we can close once implementation is verified. I don't think it needs separate approval.

[jpiesing]

March 5th meeting "One or more Beta test selected" is a placeholder but is good enough. Had planned to have a definition of "Beta" in a document that's linked to from here. Proposal

• Create a Wiki page for the explanation of what Beta is
• Link to that page from this pull request
• Accept pull request
• Create new Documentation issue to populate that page

[FritzHeiden]

For each tag there is now a filter button which disables any test in the list that does not have the corresponding tag. You can choose any combination of filters and apply the subsets on the left to them. E.g. select the released filter and click the HbbTV local button to only select tests from the HbbTV local subset that are tagged as released

[jpiesing]

That looks good to me.

[FritzHeiden]

The tests for released and beta are specified in the test-subsets.json

@jpiesing Right now only 12.5-25-50 family is included. Should I add the other families aswell?