search

canonical / checkbox

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
https://checkbox.readthedocs.io
GNU General Public License v3.0
34 stars 50 forks source link

When resuming a session and asking to re-run the last job, Checkbox does not re-run it if its outcome has been set already #428

Open pieqq opened 1 year ago

pieqq commented 1 year ago

Bug Description

When resuming a test session, if the last run job has already an outcome and comments set, the re-run choice does not work and just continues to the next job.

Two ways of fixing the issue:

  1. Either we decide that a job with outcome (and optionally comments) already set is good to go, in which case we should not display the following menu at all and we should resume to the next job directly:
What do you want to do with that job?
  c => add comment
  s => skip that job
  p => mark it as passed and continue
  f => mark it as failed and continue
  r => run it again
  1. Either we decide we should leave it up to the tester to decide what to do, in which case re-run should do what it advertises :)

To Reproduce

  1. Run any test plan
  2. When available, select q => save the session and quit (this means that the last job run has had a outcome set already)
  3. Restart checkbox-cli
  4. Select r => resume this session. A message should be displayed, such as Previous session run tried to execute job: com.canonical.certification::smoke/user-interact
  5. Select r => run it again

Expected output

The job mentioned in step 4 is re-run.

Actual output

The job mentioned in step 4 is not re-run; instead, the next job is started.

Environment

Checkbox v2.4 (local mode)

Relevant log output

I start a session, running a modified smoke test plan. When the manual job is run, I press `Ctrl+C` to exit Checkbox:

$ checkbox-cli
$PROVIDERPATH is defined, so following provider sources are ignored ['/usr/local/share/plainbox-providers-1', '/usr/share/plainbox-providers-1', '/home/pieq/.local/share/plainbox-providers-1', '/var/tmp/checkbox-providers-develop'] 
Preparing...
Reports will be saved to: /home/pieq/.local/share/checkbox-ng
--------------[ Running job 1 / 4. Estimated time left: unknown ]---------------
-----------------------------[ smoke/uiv-blocker ]------------------------------
ID: com.canonical.certification::smoke/uiv-blocker
Category: com.canonical.plainbox::uncategorised
Test user-interact job
Pick an action
    => press ENTER to continue
  c => add a comment
  s => skip this job
  q => save the session and quit
[csq]: 
... 8< -------------------------------------------------------------------------
------------------------------------------------------------------------- >8 ---
Outcome: job needs verification
Please decide what to do next:
  outcome: job needs verification
  comments: none
Pick an action
  c => add a comment
  p => set outcome to pass
  f => set outcome to fail
  s => set outcome to skip
  r => re-run this job
    => set suggested outcome [job passed]
[cpfsr]: p
--------------[ Running job 2 / 4. Estimated time left: unknown ]---------------
--------------------------------[ smoke/manual ]--------------------------------
ID: com.canonical.certification::smoke/manual
Category: com.canonical.plainbox::uncategorised
Purpose:

This test checks that the manual plugin works fine

Steps:

1. Add a comment
    2. Set the result as passed

Outcome: job needs verification
Verification:

Check that in the report the result is passed and the comment is displayed

Please decide what to do next:
  outcome: job needs verification
  comments: none
Pick an action
  c => add a comment
  p => set outcome to pass
  f => set outcome to fail
  s => set outcome to skip
[cpfs]: ^C⏎

I restart Checkbox and resume the session. When asked, I select to re-run the job, and it works as expected (because the outcome of the last run job, smoke/manual, was not set since I exited Checkbox beforehands):

$ checkbox-cli
$PROVIDERPATH is defined, so following provider sources are ignored ['/usr/local/share/plainbox-providers-1', '/usr/share/plainbox-providers-1', '/home/pieq/.local/share/plainbox-providers-1', '/var/tmp/checkbox-providers-develop'] 
=========================[ Resume Incomplete Session ]==========================
There is 1 incomplete session that might be resumed
Do you want to resume session 'session_title-2023-04-17T07.19.59'?
  r => resume this session
  n => next session
  c => create new session
  d => delete old sessions
[rncd]: r
Previous session run tried to execute job: com.canonical.certification::smoke/manual
What do you want to do with that job?
  c => add comment
  s => skip that job
  p => mark it as passed and continue
  f => mark it as failed and continue
  r => run it again
[cspfr]: r
Reports will be saved to: /home/pieq/.local/share/checkbox-ng
--------------[ Running job 1 / 3. Estimated time left: unknown ]---------------
--------------------------------[ smoke/manual ]--------------------------------
ID: com.canonical.certification::smoke/manual
Category: com.canonical.plainbox::uncategorised
Purpose:

This test checks that the manual plugin works fine

Steps:

1. Add a comment
    2. Set the result as passed

Outcome: job needs verification
Verification:

Check that in the report the result is passed and the comment is displayed

Please decide what to do next:
  outcome: job needs verification
  comments: none
Pick an action
  c => add a comment
  p => set outcome to pass
  f => set outcome to fail
  s => set outcome to skip
[cpfs]: p

Now, I save the session and quit. This time, the last executed job is still smoke/manual, but its outcome has been set to pass:

--------------[ Running job 2 / 3. Estimated time left: unknown ]---------------
----------------------------[ smoke/user-interact ]-----------------------------
ID: com.canonical.certification::smoke/user-interact
Category: com.canonical.plainbox::uncategorised
Test user-interact job
Pick an action
    => press ENTER to continue
  c => add a comment
  s => skip this job
  q => save the session and quit
[csq]: q

This time, when I restart Checkbox, resume the session and select re-run, Checkbox moves on to the next job (smoke/user-interact) instead of actually re-running smoke/manual:

$ checkbox-cli
$PROVIDERPATH is defined, so following provider sources are ignored ['/usr/local/share/plainbox-providers-1', '/usr/share/plainbox-providers-1', '/home/pieq/.local/share/plainbox-providers-1', '/var/tmp/checkbox-providers-develop'] 
=========================[ Resume Incomplete Session ]==========================
There is 1 incomplete session that might be resumed
Do you want to resume session 'session_title-2023-04-17T07.19.59'?
  r => resume this session
  n => next session
  c => create new session
  d => delete old sessions
[rncd]: r
Previous session run tried to execute job: com.canonical.certification::smoke/manual
What do you want to do with that job?
  c => add comment
  s => skip that job
  p => mark it as passed and continue
  f => mark it as failed and continue
  r => run it again
[cspfr]: r
Reports will be saved to: /home/pieq/.local/share/checkbox-ng
--------------[ Running job 1 / 2. Estimated time left: unknown ]---------------
----------------------------[ smoke/user-interact ]-----------------------------
ID: com.canonical.certification::smoke/user-interact
Category: com.canonical.plainbox::uncategorised
Test user-interact job
Pick an action
    => press ENTER to continue
  c => add a comment
  s => skip this job
  q => save the session and quit
[csq]: 


### Additional context

_No response_
syncronize-issues-to-jira[bot] commented 8 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1287.

This message was autogenerated

kissiel commented 8 months ago

@Hook25 I think you already fixed that. Would like you to confirm that it indeed is fixed.