disable timeout during consume_preamble_events

[alekla01]

This pr disables timeout in host_test_default during setup (allocation, flashing, etc..).

• disable timeout during setup (consume_preamble_events=True)
• enable timeout during testing (consume_preamble_events=False)

related: MBEDOSTEST-41

[coveralls]

Coverage increased (+0.02%) to 43.743% when pulling 6dcd4b920b0d69ac75afe1a1dd7df3cfee3f11ff on alekla01:update_timeout into 51edce4f3e3f32b254678f4640c4ec30844c2a7a on ARMmbed:master.

[jupe]

Please add more description why this changes - what was the problem and how this tries to fix it.

[alekla01]

master:

• problem: run_test sends host_test_finished event to conn_process after timeout_duration (~polling_timeout) if preamble (setup phase of conn_process) has not finished in timeout_duration. run_test joins conn_process. conn_process receives host_test_finished after preamble (allocate, flash, sync, etc) and exits (even if nothing is wrong).
• other: Each preamble component should have their own timeouts if required (such as polling timeout, or allocation timeout). if something goes wrong in conn_process during preamble, conn_process exits with __notify_conn_lost event, run_test receives __notify_conn_lost event and exits the while loop. run_test cannot (at least currently) realistically terminate conn_process (without somewhat significant problems, related to such as allocation and flashing).

-> execute
|| parallel

0 sec: run_test  -> start process conn_process()
0 sec: run_test  -> while timeout_duration: handle events || conn_process -> preamble(allocate, flash, sync, etc..)
60 sec: run_test -> timeout, send __host_test_finished    || conn_process -> preamble
60 sec: run_test -> join(conn_process)                    || conn_process -> preamble
80 sec: run_test -> joined                                || conn_process -> preamble
90 sec: run_test -> joined                                || conn_process -> preamble done, ready to begin testing.
90 sec: run_test -> joined                                || conn_process -> oh, received __host_test_finished, well lets not run the tests then.
90 sec: run_test -> result=timeout                        || conn_process -> exit

---

solution:

• what: Remove/disable timeout during preamble.
• how: in the (run_test) while loop only check timeout_duration if consume_preamble_events=False, i.e., executing tests, otherwise (consume_preamble_events=True) wait for further events (such as __host_test_name, which sets consume_preamble_events=False) from conn_process.

-> execute
|| parallel

0 sec: run_test   -> start process conn_process()
0 sec: run_test   -> handle events                         || conn_process -> preamble(allocate, flash, sync, etc..)
60 sec: run_test  -> handle events                         || conn_process -> preamble
80 sec: run_test  -> handle events                         || conn_process -> preamble
90 sec: run_test  -> handle events                         || conn_process -> preamble done, ready to begin testing.
90 sec: run_test  -> timeout_duration=test_case_timeout    || conn_process -> begin_testing
90 sec: run_test  -> while timeout_duration: handle events || conn_process -> testing, send events
100 sec: run_test -> result=test_result                    || conn_process -> tests executed, exit

The solution is not ideal, but should work better than the current implementation.

[theotherjimmy]

@bridadan Does this seem like an okay fix to you. I'm not sure I'm wiling to drop a timeout...

[bridadan]

I don't think removing the timeout is the right fix. If the process hangs indefinitely it could seriously hurt CI processes.

From the JIRA issue you mentioned, I suggested just increasing the polling timeout (-P or --polling-timeout). Would that not be enough in this situation?