Speed up process for a developer iteratively debugging a single failing test

[jpiesing]

(Edited to emphasise the different roles) One scenario where the tests will be used is with a QA engineer running the whole set of tests and passing the failing tests to a developer to fix. In this scenario, the developer gets a report of a failure and keeps repeatedly running that one failing test while debugging the implementation. In these circumstances, the elapsed time to run the test and obtain the result is very important. Concerns have been raised that this elapsed time is too long with the test runner + OF as currently used.

One possibility would be what we discussed previously, the camera streams to the test runner and the OF starts processing that streamed data incrementally as the data arrives without waiting for the end of a test or indeed the end of a test run.

Other possibilities may exist.

[gitwjr]

The following may already be in place. I haven't run it in many months. Would it be possible for the TR/OF to pause as soon as a failure is detected to let the developer choose to analyze the result right then (e.g. look at the failure and choose to rerun just that test after analysis/making a change) or click "continue" to the end and then choose to rerun just the failed test? It could be a selection at the start "pause on failure" or "run to end". To avoid the test run "stalling" if the person has walked away perhaps an "alarm" could sound when it pauses. If the engineer does not respond after X seconds, the run continues to either the next failure or the end and then lets them rerun a selected failed test. We could also offer an easy way to choose to skip a particular failed test and rerunning others if the developer believes they understand why that test failed, or the failure is unimportant to their application, or is working on other failures. or needs to wait for a new software build that fixes it. Any reports from "skipping a test" would need to indicate it was skipped.

If the camera stream was processed in parallel as well, this might make the iterative process much faster.

[jpiesing]

The following may already be in place. I haven't run it in many months. Would it be possible for the TR/OF to pause as soon as a failure is detected to let the developer choose to analyze the result right then (e.g. look at the failure and choose to rerun just that test after analysis/making a change) or click "continue" to the end and then choose to rerun just the failed test?

Sadly that probably won't help very much if either;

• The developer is only (re-)running one test that a QA person reported as failing
• The failure is one detected by the OF which means the recording has to be downloaded from the camera first

It could be a selection at the start "pause on failure" or "run to end". To avoid the test run "stalling" if the person has walked away perhaps an "alarm" could sound when it pauses. If the engineer does not respond after X seconds, the run continues to either the next failure or the end and then lets them rerun a selected failed test.

I think this is confusing the roles of the QA engineer and the developer. If the developer is investigating a failure reported by a QA engineer then they probably have only one test that they're repeatedly re-running while they change the code each time.

We could also offer an easy way to choose to skip a particular failed test and rerunning others if the developer believes they understand why that test failed, or the failure is unimportant to their application, or is working on other failures. or needs to wait for a new software build that fixes it. Any reports from "skipping a test" would need to indicate it was skipped.

IMHO What you describe is appropriate for the QA person but not the developer.

If the camera stream was processed in parallel as well, this might make the iterative process much faster.