canonical / checkbox

Checkbox
https://checkbox.readthedocs.io
GNU General Public License v3.0
30 stars 44 forks source link

Fix and test unpickleable case (bugfix) #1271

Closed Hook25 closed 3 weeks ago

Hook25 commented 3 weeks ago

Description

When the timeout decorator is used to time out the bluetooth test, the process seems to hang forever after raising an exception. The reason is that the exception is a custom un-picklable exception, so the decorator is unable to propagate it up, crashing silently (as it is doing so in a separate thread) when trying to push it (as we were using a Queue and not a SimpleQueue). Subsequently the other process detects that the child is dead but not finding anything in the exception queue (as the child died while trying to push to it) it hangs waiting for a result forever. This tests and fixes this issue (note that removing the fix, the test will fail 100% of the times)

Resolved issues

Fixes: CHECKBOX-1449 Fixes: https://github.com/canonical/checkbox/issues/1263

Documentation

Added comments to explain what is going on and why things are done the way they are

Tests

This adds 2 test cases for this new situation

Also tested on device: Reproduce:

$ snap install checkbox20 --beta
$ snap install checkbox --channel=20.04/beta
$ checkbox.checkbox-cli run com.canonical.certification::device  com.canonical.certification::bluetooth4/beacon_eddystone_url_hci0
[...]
==============[ Running job 1 / 1. Estimated time left: 0:00:10 ]===============
[ Test system can get beacon EddyStone URL advertisements on the hci0 adapter ]-
ID: com.canonical.certification::bluetooth4/beacon_eddystone_url_hci0
Category: com.canonical.plainbox::bluetooth
... 8< -------------------------------------------------------------------------
Enter sudo password:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/snap/checkbox20/current/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/snap/checkbox20/current/lib/python3.8/site-packages/checkbox_support/vendor/beacontools/scanner.py", line 148, in run
    self.hci_version = self.get_hci_version()
  File "/snap/checkbox20/current/lib/python3.8/site-packages/checkbox_support/vendor/beacontools/scanner.py", line 173, in get_hci_version
    resp = self.backend.send_req(self.socket, OGF_INFO_PARAM, OCF_READ_LOCAL_VERSION,
  File "/snap/checkbox20/current/lib/python3.8/site-packages/checkbox_support/vendor/beacontools/backend/linux.py", line 23, in send_req
    return bluez.hci_send_req(socket, group_field, command_field, event, rlen, params, timeout)
_bluetooth.error: (100, 'Network is down')
Function failed but the timeout decorator is unable to propagate this un-picklable exception:
_bluetooth.error: (100, 'Network is down')

// process hangs forever

Fix it live:

$ snap install overlay
$ /snap/overlay/current/overlay /snap/checkbox20/current/
$ sudo vim /snap/checkbox20/current/lib/python3.8/site-packages/checkbox_support/helpers/timeout.py

// replace this file with the one in this PR

$ checkbox.checkbox-cli run com.canonical.certification::device  com.canonical.certification::bluetooth4/beacon_eddystone_url_hci0
[...]
==============[ Running job 1 / 1. Estimated time left: 0:00:10 ]===============
[ Test system can get beacon EddyStone URL advertisements on the hci0 adapter ]-
ID: com.canonical.certification::bluetooth4/beacon_eddystone_url_hci0
Category: com.canonical.plainbox::bluetooth
... 8< -------------------------------------------------------------------------
Enter sudo password:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/snap/checkbox20/current/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/snap/checkbox20/current/lib/python3.8/site-packages/checkbox_support/vendor/beacontools/scanner.py", line 148, in run
    self.hci_version = self.get_hci_version()
  File "/snap/checkbox20/current/lib/python3.8/site-packages/checkbox_support/vendor/beacontools/scanner.py", line 173, in get_hci_version
    resp = self.backend.send_req(self.socket, OGF_INFO_PARAM, OCF_READ_LOCAL_VERSION,
  File "/snap/checkbox20/current/lib/python3.8/site-packages/checkbox_support/vendor/beacontools/backend/linux.py", line 23, in send_req
    return bluez.hci_send_req(socket, group_field, command_field, event, rlen, params, timeout)
_bluetooth.error: (100, 'Network is down')
Function failed but the timeout decorator is unable to propagate this un-picklable exception:
_bluetooth.error: (100, 'Network is down')

------------------------------------------------------------------------- >8 ---
Outcome: job failed
[...] // session goes on as normal
codecov[bot] commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 44.06%. Comparing base (8597b08) to head (16f0f07). Report is 3 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1271 +/- ## ========================================== + Coverage 44.04% 44.06% +0.01% ========================================== Files 358 358 Lines 38747 38751 +4 Branches 6569 6569 ========================================== + Hits 17065 17074 +9 + Misses 21019 21014 -5 Partials 663 663 ``` | [Flag](https://app.codecov.io/gh/canonical/checkbox/pull/1271/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | Coverage Δ | | |---|---|---| | [checkbox-support](https://app.codecov.io/gh/canonical/checkbox/pull/1271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | `52.36% <100.00%> (+0.14%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.