search

canonical / checkbox

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
https://checkbox.readthedocs.io
GNU General Public License v3.0
34 stars 50 forks source link

LP1954865: The sequence of job execution is different between first run and rerun #91

Open beliaev-maksim opened 2 years ago

beliaev-maksim commented 2 years ago

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1954865

Summary

Status Created on Heat Importance Security related
Confirmed 2021-12-15 03:23:18 6 High False

Description

[Summary] When user reruns the failed jobs, the execution order will lose the original sequence.

[Steps to reproduce]

  1. Run checkbox-cli
  2. Choose "USB3 test (Manual)" plan and execute it
  3. Plug USB 2.0 to make job "usb3/insert" be failed
  4. Rerun all cases (usb3/insert, usb3/storage-automated and usb3/remove)

[Expected result] The order should be

  1. usb3/insert
  2. usb3/storage-automated
  3. usb3/remove

[Actual result] Rerun sequence will be

  1. usb3/insert
  2. usb3/remove
  3. usb3/storage-automated

[Failure rate] 3/3

Attachments

No attachments

Tags: []

syncronize-issues-to-jira[bot] commented 10 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1120.

This message was autogenerated

fernando79513 commented 10 months ago

I repeated the test and the issue seems to be present still. The order is not kept when a test fails.

fernando79513 commented 2 months ago

Initially, we discussed renaming the output file of the insertion test to avoid ordering issues. However, I’m considering a different approach: joining the three separate jobs (insertion, storage, and removal) into one. This would eliminate the ordering problem and also resolve the inconsistent job execution sequences between runs.

Currently, our test plans run all three tests sequentially. We could also create a version that only performs insertion and removal without storage if needed.

The idea is to have these tests:

Insertion + Storage + Removal Insertion + Removal (Insertion maybe?) That will force the insertion test to always be run before the storage and the removal, but that is currently the case, as both of them depend on the insertion test. This would only force them to be run all of them at the same time.