search

cta-wave / device-observation-framework

This repository contains the WAVE Device Observation Framework. The DPCTF Device Observation Framework determines pass or fail results, based on observations taken of tests which are run on a device by the DPCTF Test Runner. DPCTF is the Device Playback Compatibility Task Force of the CTA WAVE Project (https://CTA.tech/WAVE and standards@CTA.tech for more info).
Other
1 stars 3 forks source link

Unable to identify test #72

Closed FritzHeiden closed 8 months ago

FritzHeiden commented 8 months ago

Running the OF on one of the recordings results in this error:

Exception: Unable to identify the test. Observation will not be made.

I can see from the log that it correctly detects the test from the "Next test about to start" page, then 10 seconds later crashes with the mentioned error. In the recording, checking the frame counts from the logs, I cannot see anything problematic. I use the current config.ini from the repo, just set the test_runner_url.

Logs ``` Processed to frame 34800... Processed to frame 34810... INFO Start a New test: cfhd_12.5_25_50-local/low-latency-short-buffer-playback__t2.html Processed to frame 34820... Processed to frame 34830... Processed to frame 34840... Processed to frame 34850... Processed to frame 34860... Processed to frame 34870... Processed to frame 34880... Processed to frame 34890... Processed to frame 34900... Processed to frame 34910... Processed to frame 34920... Processed to frame 34930... Processed to frame 34940... Processed to frame 34950... Processed to frame 34960... Processed to frame 34970... Processed to frame 34980... Processed to frame 34990... Processed to frame 35000... Processed to frame 35010... Processed to frame 35020... Processed to frame 35030... Processed to frame 35040... Processed to frame 35050... Processed to frame 35060... Processed to frame 35070... Processed to frame 35080... Processed to frame 35090... Processed to frame 35100... Processed to frame 35110... Processed to frame 35120... Processed to frame 35130... Processed to frame 35140... Processed to frame 35150... Processed to frame 35160... Processed to frame 35170... Processed to frame 35180... Processed to frame 35190... Processed to frame 35200... Processed to frame 35210... Processed to frame 35220... Processed to frame 35230... Processed to frame 35240... Processed to frame 35250... Processed to frame 35260... Processed to frame 35270... Processed to frame 35280... Processed to frame 35290... Processed to frame 35300... Processed to frame 35310... Processed to frame 35320... Processed to frame 35330... Processed to frame 35340... Processed to frame 35350... Processed to frame 35360... Processed to frame 35370... Processed to frame 35380... Processed to frame 35390... Processed to frame 35400... Processed to frame 35410... Processed to frame 35420... Processed to frame 35430... Processed to frame 35440... Processed to frame 35450... Processed to frame 35460... Processed to frame 35470... Processed to frame 35480... Processed to frame 35490... Processed to frame 35500... Processed to frame 35510... Processed to frame 35520... Processed to frame 35530... Processed to frame 35540... Processed to frame 35550... Processed to frame 35560... Processed to frame 35570... Processed to frame 35580... Processed to frame 35590... Processed to frame 35600... Processed to frame 35610... Processed to frame 35620... Processed to frame 35630... Processed to frame 35640... Processed to frame 35650... Processed to frame 35660... Processed to frame 35670... Processed to frame 35680... Processed to frame 35690... Processed to frame 35700... Processed to frame 35710... Processed to frame 35720... Processed to frame 35730... Processed to frame 35740... Processed to frame 35750... Processed to frame 35760... Processed to frame 35770... Processed to frame 35780... Processed to frame 35790... Processed to frame 35800... Processed to frame 35810... Processed to frame 35820... Processed to frame 35830... Processed to frame 35840... Processed to frame 35850... Processed to frame 35860... Processed to frame 35870... Processed to frame 35880... Processed to frame 35890... Processed to frame 35900... Processed to frame 35910... Processed to frame 35920... Processed to frame 35930... Processed to frame 35940... Processed to frame 35950... Processed to frame 35960... Processed to frame 35970... Processed to frame 35980... Processed to frame 35990... Processed to frame 36000... Processed to frame 36010... Processed to frame 36020... Processed to frame 36030... Processed to frame 36040... Processed to frame 36050... Processed to frame 36060... Processed to frame 36070... Processed to frame 36080... Processed to frame 36090... Processed to frame 36100... Processed to frame 36110... Processed to frame 36120... Processed to frame 36130... Processed to frame 36140... Processed to frame 36150... Processed to frame 36160... Processed to frame 36170... Processed to frame 36180... Processed to frame 36190... Processed to frame 36200... Processed to frame 36210... Processed to frame 36220... Processed to frame 36230... Processed to frame 36240... Processed to frame 36250... Processed to frame 36260... Processed to frame 36270... Processed to frame 36280... Processed to frame 36290... INFO Making observation [OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order.... INFO Making observation [OF] Video: The playback duration shall match the duration of the CMAF Track... INFO Making observation [OF] Video: The presented sample shall match the one reported by the currentTime value within the tolerance of +/-(2/framerate + 20ms)... WARNING Mezzanine QR code is detected before identifying the test. observations won't be made, stop process if you want. Processed to frame 36300... WARNING Mezzanine QR code is detected before identifying the test. observations won't be made, stop process if you want. WARNING Mezzanine QR code is detected before identifying the test. observations won't be made, stop process if you want. Processed to frame 36310... WARNING Mezzanine QR code is detected before identifying the test. observations won't be made, stop process if you want. WARNING Mezzanine QR code is detected before identifying the test. observations won't be made, stop process if you want. Processed to frame 36320... WARNING Mezzanine QR code is detected before identifying the test. observations won't be made, stop process if you want. WARNING Mezzanine QR code is detected before identifying the test. observations won't be made, stop process if you want. WARNING Test status QR code is detected before identifying the test. observations won't be made, stop process if you want. ERROR Serious error is detected! Unable to identify the test. Observation will not be made.: Traceback (most recent call last): File "observation_framework.py", line 589, in main run( File "observation_framework.py", line 445, in run last_camera_frame_number = observation_framework.iter_qr_codes_in_video( File "/usr/app/device-observation-framework/observation_framework_processor.py", line 745, in iter_qr_codes_in_video self._process_test_status_qr_code(new_test_status_qr_code) File "/usr/app/device-observation-framework/observation_framework_processor.py", line 520, in _process_test_status_qr_code self._make_observations() File "/usr/app/device-observation-framework/observation_framework_processor.py", line 456, in _make_observations raise Exception( Exception: Unable to identify the test. Observation will not be made. Traceback (most recent call last): File "observation_framework.py", line 589, in main run( File "observation_framework.py", line 445, in run last_camera_frame_number = observation_framework.iter_qr_codes_in_video( File "/usr/app/device-observation-framework/observation_framework_processor.py", line 745, in iter_qr_codes_in_video self._process_test_status_qr_code(new_test_status_qr_code) File "/usr/app/device-observation-framework/observation_framework_processor.py", line 520, in _process_test_status_qr_code self._make_observations() File "/usr/app/device-observation-framework/observation_framework_processor.py", line 456, in _make_observations raise Exception( Exception: Unable to identify the test. Observation will not be made. ```
yanj-github commented 8 months ago

@FritzHeiden thanks for reporting. Can you share the recording please? Or can you check from frame 36290 to 36300 if there is any pre-test QR code please? And check if it is clear enough to be detected by OF. It seems that OF detected content playing without detecting pre-test QR code. cfhd_12.5_25_50-local/low-latency-short-buffer-playback__t2.html is completed to this is the next test.

FritzHeiden commented 8 months ago

Pre-Test at around 34820: image

Video playback starts until an error occurs during test execution at around 36290 and next test is executed before video is played through: image

Next pre-test shows at around 37200.

yanj-github commented 8 months ago

Thanks @FritzHeiden Can you confirm the following please? I guess test run shows twice of status = finished. OF seems tried to make observation twice for the same test. OF determines end of test by status = finished.

FritzHeiden commented 8 months ago

It first shows status: error, then goes over to status: finished, after which it redirects to the next test.

yanj-github commented 8 months ago

It first shows status: error, then goes over to status: finished, after which it redirects to the next test.

Processed to frame 36290... Is there status = finished here? And more Mezzanine QR code is detected after status = finished? Processed to frame 36310... And more Mezzanine QR code is detected after status = finished? Processed to frame 36320... Is there another status = finished (this is the cause of exception) before the next pre-test QR code at 37200?

If it is easy can you kindly run OF again with --log debug and send me log files in logs folder please?

FritzHeiden commented 8 months ago

The video continues to play for a few seconds with the QR code being updated on time update event, so there are multiple QR code updates containing status: finished, yes.

If it is easy can you kindly run OF again with --log debug and send me log files in logs folder please?

Yes I can do that, will take some time to get there though. I sent you the recording if that helps.

yanj-github commented 8 months ago

@FritzHeiden Thanks, I just got the recording file that I had a look. The cause of the issue is this: image Status = finished but the content is still playing.

OF is unable to determine this is error in the same test or multiple tests, as it detects end of test by Status = finished. Is there alternative way to signal end of current test to OF?

Previously we suggested test runner to add end of test QR code at the end of current test the same way we have pre-test QR code, how difficult is this can be done please?

FritzHeiden commented 8 months ago

An easy and quick fix would be to just stop updating the QR code once status = finished.

As for a solution for this recording, wouldn't it make sense to not abort observation with critical error but instead keep looking for new test (like its done at the very beginning of the recording)?

yanj-github commented 8 months ago

@FritzHeiden yep agree, it would be a quick fix, can be done easy. However there is a risk of some tests in the middle failed quietly without notification when checking the results in test session some test has no OF results. I will print out warning instead on the OF log for such a case.

yanj-github commented 8 months ago

@FritzHeiden this is now done, can you check please?

FritzHeiden commented 8 months ago

Thanks @yanj-github I got past that point in the analysis now, however I get a new error at 126910:

logs ``` Processed to frame 126890... Processed to frame 126900... Processed to frame 126910... INFO Making observation [OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order.... INFO Making observation [OF] Video: The playback duration shall match the duration of the CMAF Track... ERROR Serious error is detected! list index out of range: Traceback (most recent call last): File "observation_framework.py", line 589, in main run( File "observation_framework.py", line 445, in run last_camera_frame_number = observation_framework.iter_qr_codes_in_video( File "/usr/app/device-observation-framework/observation_framework_processor.py", line 745, in iter_qr_codes_in_video self._process_test_status_qr_code(new_test_status_qr_code) File "/usr/app/device-observation-framework/observation_framework_processor.py", line 520, in _process_test_status_qr_code self._make_observations() File "/usr/app/device-observation-framework/observation_framework_processor.py", line 434, in _make_observations results = self.test_class.make_observations( File "/usr/app/device-observation-framework/test_code/sequential_track_playback.py", line 367, in make_observations result, updated_audio_segments = observation_class.make_observation( File "/usr/app/device-observation-framework/observations/duration_matches_cmaf_track.py", line 310, in make_observation mezzanine_qr_codes[: change_starting_index_list[1] - 1], IndexError: list index out of range Traceback (most recent call last): File "observation_framework.py", line 589, in main run( File "observation_framework.py", line 445, in run last_camera_frame_number = observation_framework.iter_qr_codes_in_video( File "/usr/app/device-observation-framework/observation_framework_processor.py", line 745, in iter_qr_codes_in_video self._process_test_status_qr_code(new_test_status_qr_code) File "/usr/app/device-observation-framework/observation_framework_processor.py", line 520, in _process_test_status_qr_code self._make_observations() File "/usr/app/device-observation-framework/observation_framework_processor.py", line 434, in _make_observations results = self.test_class.make_observations( File "/usr/app/device-observation-framework/test_code/sequential_track_playback.py", line 367, in make_observations result, updated_audio_segments = observation_class.make_observation( File "/usr/app/device-observation-framework/observations/duration_matches_cmaf_track.py", line 310, in make_observation mezzanine_qr_codes[: change_starting_index_list[1] - 1], IndexError: list index out of range ```
yanj-github commented 8 months ago

@FritzHeiden Is this truncated test? I have looked at the recording the truncated test has main and ad with same content ID A1. OF won't be able to detect the truncating position from mezzanine if they are using same content ID.

FritzHeiden commented 8 months ago

@yanj-github Yes I can see the same from the recording. As other devices played this properly this has to be some issue with the device. Therefore, this should be something the OF should detect as an error. What do you think?

yanj-github commented 8 months ago

@yanj-github Yes I can see the same from the recording. As other devices played this properly this has to be some issue with the device. Therefore, this should be something the OF should detect as an error. What do you think?

If this happens commonly due to device issues I will unblock the process from OF.

FritzHeiden commented 8 months ago

I have another recording where the same error seems to occur:

logs ``` Processed to frame 122640... Processed to frame 122650... Processed to frame 122660... INFO Making observation [OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order.... INFO Making observation [OF] Video: The playback duration shall match the duration of the CMAF Track... ERROR Serious error is detected! list index out of range: Traceback (most recent call last): File "observation_framework.py", line 589, in main run( File "observation_framework.py", line 445, in run last_camera_frame_number = observation_framework.iter_qr_codes_in_video( File "/usr/app/device-observation-framework/observation_framework_processor.py", line 745, in iter_qr_codes_in_video self._process_test_status_qr_code(new_test_status_qr_code) File "/usr/app/device-observation-framework/observation_framework_processor.py", line 520, in _process_test_status_qr_code self._make_observations() File "/usr/app/device-observation-framework/observation_framework_processor.py", line 434, in _make_observations results = self.test_class.make_observations( File "/usr/app/device-observation-framework/test_code/sequential_track_playback.py", line 367, in make_observations result, updated_audio_segments = observation_class.make_observation( File "/usr/app/device-observation-framework/observations/duration_matches_cmaf_track.py", line 310, in make_observation mezzanine_qr_codes[: change_starting_index_list[1] - 1], IndexError: list index out of range Traceback (most recent call last): File "observation_framework.py", line 589, in main run( File "observation_framework.py", line 445, in run last_camera_frame_number = observation_framework.iter_qr_codes_in_video( File "/usr/app/device-observation-framework/observation_framework_processor.py", line 745, in iter_qr_codes_in_video self._process_test_status_qr_code(new_test_status_qr_code) File "/usr/app/device-observation-framework/observation_framework_processor.py", line 520, in _process_test_status_qr_code self._make_observations() File "/usr/app/device-observation-framework/observation_framework_processor.py", line 434, in _make_observations results = self.test_class.make_observations( File "/usr/app/device-observation-framework/test_code/sequential_track_playback.py", line 367, in make_observations result, updated_audio_segments = observation_class.make_observation( File "/usr/app/device-observation-framework/observations/duration_matches_cmaf_track.py", line 310, in make_observation mezzanine_qr_codes[: change_starting_index_list[1] - 1], IndexError: list index out of range ```

I will send this recording to you aswell

yanj-github commented 8 months ago

@FritzHeiden truncated test is now fixed.

FritzHeiden commented 8 months ago

@FritzHeiden truncated test is now fixed.

The analysis now finished for the first video. Thanks. Will try the second now

FritzHeiden commented 8 months ago

Second recording ran through without problems aswell. Closing.