documentation on debugging observation failures

[jpiesing]

Please consider writing some documentation on debugging observation failures. Perhaps a general section on what's common to all observations and then a paragraph on each observation.

For example, if failing the observation for duration with an error message like "Playback duration 30156.67ms does not match expected duration 30000.0ms +/- tolerance of 20ms. Allowed tolerance is20ms and duration frame tolerance is 0. Starting missing frame number is 0. Starting frames to take out is 0. Endingmissing frame number is 0."

I guess to debug this, someone would need to look at the recording and identify the start and end frames used to compute the duration and calculate the time difference between those frames themselves. If frames are being duplicated or dropped then that would show up as a failure of "The presented sample matches the one reported by the currentTime value within the tolerance of the sample dur…" with the first frame reported in that error message being the one that is duplicated or the first one after the one that has been dropped.

[jpiesing]

Another example;

[OF] The presented sample matches the one reported by the currentTime value within the tolerance of the sample dur… FAIL Xx01: Allowed tolerance is 2 frames, 20ms. Time diff erence between Test Runner reported media currentTime and actual mediatime exceeded tolerance for following events: currentTime=0.0 time_diff =1.7976931348623157e+308; currentTime=0.0time_diff =1.7976931348623157e+308; currentTime=0.0 time_diff =1.7976931348623157e+308; currentTime=0.0time_diff =1.7976931348623157e+308; currentTime=7.946 time_diff =74.0;

The first of these are issue #48 but documentation giving guidance on how to investigate the last one would be helpful.

[yanj-github]

Observation-Algorithms is documented here which can be used for debugging. https://github.com/cta-wave/device-observation-framework/wiki/Observation-Algorithms

OF should be run to enable debug log "--log debug" so that "qr_code_list.csv" file is logged which contains full QR code information

[yanj-github]

I think this is documented feel free to close if you are happy as well @jpiesing

[jpiesing]

@yanj-github I haven't closed it as I think there are things that are obvious to you and less to others. Perhaps a more complete version of the following.

The following steps may help to debug a test where the observation framework reports a failure;

• Run the observation framework with --log debug to generate the file logs//session.log
• Look through session.log to find the frame number where the test with the failing observation starts (this will be indicated by a line like "Start a New test: cfhd_12.5_25_50-local/sequential-track-playback__t3.html").
• From that point, look forwards to find the frame number where the second decoded frame is first detected
• From that point, look forwards to find the frame number where the last decoded frame of that test is first detected (many tests are 30s and, if using 25Hz video, 30s at 25Hz is 750 so look for the next instance of "Frame Number=750").
• You will now have 3 frame numbers in the recording
• Look at the recording using a video editor or video playback tool that gives you a frame accurate view of the file and lets you seek to a specific frame number and step frame by frame. (For Windows, xxxx or yyy will do this. For Linux, PiTiVi will do this).
• Find the first of your frame numbers in the recording. If you are unlucky, there may be an offset between the frame numbers used in your video tool and those reported by the OF. You may need to move backwards and forwards in the video tool to discover this offset.
• Examine the recording to determine if the observation should have failed.
• Note the following;
  • test streams start with a green frame and finish with a red frame. One common reason for failure is first and/or last frames not being visible which is very obvious with these starting and finishing frames.
  • if a decoder stalls for a few 10s or 100s of ms during playback, this will appear as a failure with the duration being reported as longer than expected by the amount of the stall
  • failure to play 12.5/15Hz content or 50Hz/60Hz content at the correct frame rate will appear as both a (much larger) duration failure and a current time failure.

[yanj-github]

Document is added here https://github.com/cta-wave/device-observation-framework/wiki/Debugging-Observation-Failures

[gitwjr]

@jpiesing will read through the document and comment. It is already posted in a repo. @gitwjr Add new issue on how a test engineer would find the document when there are failures.

[jpiesing]

I think this is as good as it's going to get for now. It can be improved later based on more experience.