Test runner reports all test results as "script error"

[rcottingham]

Test Runner displays an error when tests are run on a TV. Status is “error” with message “script error”. This occurs on all tests in all test groups.

The error is not displayed when testing on a PC.

To confirm issue is fixed, please retest using the complete test suite and review results from Observation Framework. OF version to use is available here

For further details on the issue please see: “Eurofins CTA WAVE Test Suite validation of the Observation Framework results including analysis 31 Jan 2023.xls”

[FritzHeiden]

We have to investigate this issue on a TV on our side. Can you provide details about the device you ran into this issue with?

[pshorrock]

Hi @FritzHeiden , unfortunately we are not permitted to share the TV device brand/model details. I can confirm however that we are using devices loaned from manufacturers with development firmware builds. However all HW chassis we are using should be in the market now. I hope that helps to at least explain our position.

[jpiesing]

@pshorrock Do you know if it happens on TVs from more than one brand? If it only happens on TVs from one brand and you're unable to identify that brand, even privately, to @louaybassbouss then I don't see how they stand any chance of reproducing this issue.

[gitwjr]

@pshorrock If your model is running development firmware, is it possible this was fixed for released models? Would it be possible to speak with the manufacturer to see if they are aware of it, have potentially fixed it, or to get latest production firmware to confirm the problem exists in production models?

[jpiesing]

@pshorrock If your model is running development firmware, is it possible this was fixed for released models? Would it be possible to speak with the manufacturer to see if they are aware of it, have potentially fixed it, or to get latest production firmware to confirm the problem exists in production models?

Alternatively, find a TV with production software from the same manufacturer and see if the problem also occurs there - not running all the tests just one very simple one.

[pshorrock]

Hi @jpiesing @gitwjr , we can confirm that we have checked with different (brand) devices and that we did not see the same issue which may indicate it is a local issue on the main device used and not a wider regression. Knowing however that this has not been observed / reproduced as part of others testing on devices would equally help, I'm not sure if anyone else is saying that? Are we the only entity running tests on devices? We are hoping that is not the case as we are more focused on the Observation Framework but are happy to flag our results / observations to the group.

[jpiesing]

Are we the only entity running tests on devices?

The 2021 and 2022 RfPs included the following;

The test cases need to be executed on at least 2 Android platforms, on at least 2 Smart TVs (at least one using an HbbTV browser), on at least one iPad (iPad OS version 13 or greater) and on at least one HDMI media streaming stick. Successful execution is preferred but not required if prevented by bugs or acknowledged limitations in the streaming support on the platform concerned. Any such bugs should be reported in the appropriate issue tracker if not already present.

[rcottingham]

Confirmed as an issue with the DUT (TV running non-production firmware).

Issue closed.