search

cta-wave / device-observation-framework

This repository contains the WAVE Device Observation Framework. The DPCTF Device Observation Framework determines pass or fail results, based on observations taken of tests which are run on a device by the DPCTF Test Runner. DPCTF is the Device Playback Compatibility Task Force of the CTA WAVE Project (https://CTA.tech/WAVE and standards@CTA.tech for more info).
Other
1 stars 3 forks source link

optimisation when analysing failures reported by OF - not re-running QR code detection #60

Closed jpiesing closed 1 year ago

jpiesing commented 1 year ago

When analysing failures reported by the OF, it would be really helpful if QR code detection was not re-run if there is a valid qr_code_list.csv file (valid = more recent than recording, recording has correct file size).

jpiesing commented 1 year ago

Another variation on the theme of optimisation for analysing failures would be to be able to specify which part of a recording is to be analysed by the OF, e.g. starting at a particular frame number for a certain number of frames. This would enable re-running the OF just for one test.

pshorrock commented 1 year ago

@jpiesing I have emailed separately with more detailed rationale but as a work-around to help make validation more efficient would the video content separated out to be test by test help (rather than one continuous file)? If so we have already done that for you and can share it back. As also mentioned in the email we do have some concerns adding in capability which might open up an ability to manually work around failures ("QR code detection was not re-run if there is a valid qr_code_list.csv file") where we see using the output files can help to observe necessary tolerances. And can see in the other case opencv limiting the efficiency of stepping through a video file to get to a certain point, we think simply presenting a single video file to the OF which has only the test you are looking at included would be more efficient for the OF, and much quicker for us to help support you with today to help validation. @yanj-github @rcottingham can both help to follow up on this as needed. If we have misunderstood to reasons for the request (validation of the OF and test suite, not typical functionality required by users) we are happy to talk it through. And as part of that we do see dividing up content into separate files as a relatively easy task we'd be happy to document.

jpiesing commented 1 year ago

Closing as this is less relevant once we have agreed values for tolerances and thresholds.