search

canonical / checkbox

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
https://checkbox.readthedocs.io
GNU General Public License v3.0
33 stars 50 forks source link

Cannot export the submission tar file after finishing the session when the test plan is not whole #1557

Closed weizhenwu closed 1 day ago

weizhenwu commented 4 weeks ago

Bug Description

Checkbox cannot export the submission tar file after finishing the session. The errors below will be reported and the session will stay incomplete. It happens only when I don't run the whole test plan but select partial tests.

To Reproduce

  1. install following 
    snap install checkbox22 --beta
snap install checkbox-carmel-classic --classic --edge
snap install checkbox-ce-oem --channel=22.04/edge --classic
  2. Launch checkbox checkbox-carmal-classic.checkbox-cli
  3. Select carmel-classic-automated
  4. Select only camera tests and start testing
  5. Press F to export to submission after finishing

Environment

OS: ubuntu server 22.04 Checkbox type: snap Checkbox version: 4.2.0-dev121 DUT: 202409-35561

Relevant log output

ERROR:plainbox.bug:Undeclared exception UndefinedError raised from export_to_transport
ERROR:checkbox-ng.launcher.stages:Problem with a '2_tar_file' report using 'com.canonical.plainbox::tar' exporter sent to '/home/ubuntu/.local/share/checkbox-ng/submission_2024-10-23T03.08.00.817432.tar.xz' transport. Reason 'dict object' has no attribute 'com.canonical.contrib::ce-oem-info/snapd_installation_time'
Traceback (most recent call last):
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/stages.py", line 669, in _export_results
    result = self.sa.export_to_transport(
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/decorators.py", line 153, in wrapper
    raise exc
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/decorators.py", line 145, in wrapper
    return func(*args, **kwargs)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/session/assistant.py", line 1823, in export_to_transport
    exporter.dump_from_session_manager(self._manager, exported_stream)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/exporter/tar.py", line 74, in dump_from_session_manager
    exporter.dump_from_session_manager(manager, _s)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 205, in dump_from_session_manager
    self.dump(data, stream)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 175, in dump
    self.template.stream(data).dump(stream, encoding="utf-8")
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1612, in dump
    fp.writelines(iterable)
  File "/snap/checkbox22/current/usr/lib/python3.10/tempfile.py", line 955, in writelines
    rv = file.writelines(iterable)
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1607, in <genexpr>
    iterable = (x.encode(encoding, errors) for x in self)  # type: ignore
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1656, in __next__
    return self._next()  # type: ignore
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1348, in generate
    yield self.environment.handle_exception()
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/providers/exporters/data/checkbox.json", line 115, in top-level template code
    "name": "{{ job_state_map[job_id].job.tr_summary() }}",
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 474, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'com.canonical.contrib::ce-oem-info/snapd_installation_time'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/checkbox22/current/bin/checkbox-cli", line 8, in <module>
    sys.exit(main())
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/checkbox_cli.py", line 166, in main
    return subcmd.invoked(ctx)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/subcommands.py", line 295, in invoked
    self._export_results()
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/stages.py", line 723, in _export_results
    traceback.print_tb(exc)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 53, in print_tb
    print_list(extract_tb(tb, limit=limit), file=file)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 72, in extract_tb
    return StackSummary.extract(walk_tb(tb), limit=limit)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 364, in extract
    for f, lineno in frame_gen:
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 329, in walk_tb
    yield tb.tb_frame, tb.tb_lineno
AttributeError: 'UndefinedError' object has no attribute 'tb_frame'

Additional context

No response

syncronize-issues-to-jira[bot] commented 4 weeks ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1627.

This message was autogenerated

pieqq commented 3 weeks ago

The root cause of the issue is a fix I landed 2 weeks ago (#1520).

However, looking at the information in the description, I don't really understand what's going on here:

You select the test plan client-cert-iot-server-22-04-automated, but when Checkbox tries to generate the submission archive, it fails with the following error:

jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'com.canonical.contrib::ce-oem-info/snapd_installation_time'

This job is not part of client-cert-iot-server-22-04-automated, so I'm wondering why it gets added to the list of rejected jobs... I'll need to dig a bit more into this.

pieqq commented 2 weeks ago

I was able to see this issue when running a test plan that has test cases from another provider (with a different namespace).

It looks like the root cause is that some parts of the code are using the job_id, appending the default com.canonical.certification namespace if not found, while other parts rely on full_id.

pieqq commented 1 week ago

I still cannot reproduce this issue.

I installed the following snaps:

$ sudo snap install checkbox24 --beta
$ sudo snap install checkbox --channel=24.04/beta --classic
$ sudo snap install checkbox-ce-oem --channel=24.04/beta --classic
$ snap list
Name                       Version             Rev    Tracking         Publisher             Notes
checkbox                   4.2.0-dev127        10238  24.04/beta       ce-certification-qa   classic
checkbox-ce-oem            1.0-noble           559    24.04/beta       ce-certification-qa   classic
checkbox24                 4.2.0-dev127        322    latest/beta      ce-certification-qa   -

I followed the steps in the original description:

  1. Run checkbox-ce-oem.checkbox-cli
  2. Select client-cert-iot-server-22-04-automated test plan
  3. Select only camera tests and start testing
  4. At the end of session, press F to finish the run and export to submission

It works as expected.

I also tried the same steps, but using a ce-oem test plan instead: com.canonical.contrib::ce-oem-iot-server-24-04-automated. It works as expected.

weizhenwu commented 1 week ago

Reproduced with checkbox22 + ce-oem + project checkbox installed (as the steps in the description)

OS: ubuntu server 22.04

Reproduce Steps:

  1. launch checkbox-carmal-classic.checkbox-cli
  2. run carmel-classic-automated with only spi test (ce-oem-spi/detect) selected
  3. finish the test plan, and it will generate the submission as expected but when it asked me if I want to upload, I pressed crtl+c
  4. launch checkbox-carmal-classic.checkbox-cli again, it resumes the same session. then finishing it again will get this error
ERROR:plainbox.bug:Undeclared exception UndefinedError raised from export_to_transport
ERROR:checkbox-ng.launcher.stages:Problem with a '2_tar_file' report using 'com.canonical.plainbox::tar' exporter sent to '/home/ubuntu/.local/share/checkbox-ng/submission_2024-11-13T09.32.09.447416.tar.xz' transport. Reason 'dict object' has no attribute 'com.canonical.contrib::ce-oem-info/snapd_installation_time'
Traceback (most recent call last):
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/stages.py", line 689, in _export_results
    result = self.sa.export_to_transport(
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/decorators.py", line 153, in wrapper
    raise exc
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/decorators.py", line 145, in wrapper
    return func(*args, **kwargs)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/session/assistant.py", line 1823, in export_to_transport
    exporter.dump_from_session_manager(self._manager, exported_stream)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/exporter/tar.py", line 74, in dump_from_session_manager
    exporter.dump_from_session_manager(manager, _s)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 205, in dump_from_session_manager
    self.dump(data, stream)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 175, in dump
    self.template.stream(data).dump(stream, encoding="utf-8")
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1612, in dump
    fp.writelines(iterable)
  File "/snap/checkbox22/current/usr/lib/python3.10/tempfile.py", line 955, in writelines
    rv = file.writelines(iterable)
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1607, in <genexpr>
    iterable = (x.encode(encoding, errors) for x in self)  # type: ignore
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1656, in __next__
    return self._next()  # type: ignore
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 1348, in generate
    yield self.environment.handle_exception()
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/providers/exporters/data/checkbox.json", line 115, in top-level template code
    "name": "{{ job_state_map[job_id].job.tr_summary() }}",
  File "/snap/checkbox22/current/usr/lib/python3/dist-packages/jinja2/environment.py", line 474, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'com.canonical.contrib::ce-oem-info/snapd_installation_time'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/checkbox22/current/bin/checkbox-cli", line 8, in <module>
    sys.exit(main())
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/checkbox_cli.py", line 166, in main
    return subcmd.invoked(ctx)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/subcommands.py", line 295, in invoked
    self._export_results()
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/stages.py", line 750, in _export_results
    traceback.print_tb(exc)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 53, in print_tb
    print_list(extract_tb(tb, limit=limit), file=file)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 72, in extract_tb
    return StackSummary.extract(walk_tb(tb), limit=limit)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 364, in extract
    for f, lineno in frame_gen:
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 329, in walk_tb
    yield tb.tb_frame, tb.tb_lineno
AttributeError: 'UndefinedError' object has no attribute 'tb_frame'

Snap list

$ snap list
Name                     Version        Rev    Tracking       Publisher            Notes
checkbox-carmel-classic  0.1dev         33     latest/edge    ce-certification-qa  classic
checkbox-ce-oem          1.0-jammy      584    22.04/edge     ce-certification-qa  classic
checkbox22               4.2.0-dev127   1270   latest/beta    ce-certification-qa  -
core20                   20240911       2437   latest/stable  canonical✓           base
core22                   20241001       1666   latest/stable  canonical✓           base
lxd                      5.0.3-80aeff7  29353  5.0/stable/…   canonical✓           -
snapd                    2.63           21761  latest/stable  canonical✓           snapd
baconYao commented 1 week ago

Able to reproduce this issue

Reproduce Steps

  1. Install checkbox22 and checkbox snaps
  2. Run checkbox via control like $ checkbox.checkbox-cli control <DUT IP address>
  3. Choose IoT Client Certification for Ubuntu Core 22 (Automated Tests) test plan
  4. Choose the watchdog related cases in Power Management Set
    • Before doing this, please make sure the watchdog is enabled (the value of RuntimeWatchdogSec=30 in /etc/systemd/system.conf)
  5. Execute cases

DUT Info

CID: 202404-33953 SKU: G1200 Image Type: UC22

Checkbox Info

ceqa@ubuntu:~$ snap list
Name                                     Version         Rev    Tracking       Publisher            Notes
bluez                                    5.64-6          382    22/stable      canonical✓           -
bugit                                    1.2             35     latest/stable  pieq                 devmode
checkbox                                 4.2.0-dev127    10242  uc22/beta      ce-certification-qa  devmode
checkbox-ce-oem                          1.0-jammy       575    uc22/edge      ce-certification-qa  devmode
checkbox22                               4.2.0-dev127    1270   latest/beta    ce-certification-qa  -
core22                                   20241106        1698   latest/edge    canonical✓           base
genio-test-tool                          1.0             2      latest/stable  ceqa                 devmode
mediatek-genio                           22-0.4.2        6      22/stable      canonical✓           gadget
mediatek-genio-g1200-gpu-drivers-core22  r48p0           x1     -              -                    -
mediatek-genio-kernel                    5.15.0-1034.40  21     22/stable      canonical✓           kernel
snapd                                    2.63            21761  latest/stable  canonical✓           snapd
ubuntu-frame                             147-mir2.17.2   10824  22/stable      canonical✓           -
x-test                                   3.x             20     22/stable      kiya956              devmode

I/O log

--------------------------------------------------------------------------------
Outcome: job passed
----------------------------[ Running job 36 / 39 ]-----------------------------
-----------------[ Detect the presence of a Hardware Watchdog ]-----------------
ID: com.canonical.certification::watchdog/detect
Category: Power Management tests
--------------------------------------------------------------------------------
path: /devices/platform/soc/10007000.watchdog/watchdog/watchdog0
name: watchdog0
bus: watchdog
category: WATCHDOG
driver: mtk-wdt

--------------------------------------------------------------------------------
Outcome: job passed
----------------------------[ Running job 37 / 39 ]-----------------------------
-----------[ Check if the hardware watchdog is properly configured ]------------
ID: com.canonical.certification::watchdog/systemd-config
Category: Power Management tests
--------------------------------------------------------------------------------
systemd watchdog enabled, reset timeout: 30s
watchdog.service is not active
--------------------------------------------------------------------------------
Outcome: job passed
----------------------------[ Running job 38 / 39 ]-----------------------------
----------[ Test that the watchdog module can trigger a system reset ]----------
ID: com.canonical.certification::watchdog/trigger-system-reset-auto
Category: Power Management tests
--------------------------------------------------------------------------------
Connection lost!
[Errno 104] Connection reset by peer
----------------------------[ Running job 61 / 61 ]-----------------------------
---------------------[ Post watchdog reset service check ]----------------------
ID: com.canonical.certification::watchdog/post-trigger-system-reset-auto
Category: Power Management tests
--------------------------------------------------------------------------------
Found 0 failed units
--------------------------------------------------------------------------------
Outcome: job passed
==================================[ Results ]===================================
32.0kB [00:00, 532kB/s, file=python://stdout]                                                                                                     
  job passed   : Hardware Manifest
  job passed   : Enumerate available system executables
  job passed   : Collect information about interfaces
  job passed   : Collect information about connections
  job passed   : Collect model assertions on the device
  job passed   : Collect serial assertions on the device
  job passed   : Identify what service is managing each physical network interface
  job passed   : Collect logging from the net_if_management job
  job passed   : Attach a list of PCI devices
  job passed   : Attach output of lsusb
  job passed   : Creates resource info for RTC
  job passed   : Create resource info for supported sleep states
  job failed   : Attaches an information about all parts that constituted this snap
  job passed   : Resource to detect if dmi data is present
  job passed   : Attach a copy of /sys/class/dmi/id/*
  job passed   : Attaches json dumps of installed dkms package information.
  job passed   : Collect information about hardware devices (DMI)
  job passed   : Attach PCI configuration space hex dump
  job passed   : Collect information about the CPU
job cannot be started: Attach detailed sysfs property output from udev
  job passed   : Collect information about system memory (/proc/meminfo)
  job passed   : Create resource info for environment variables
  job passed   : Attaches JSON dumps of raw DMI devices
  job passed   : Attach the contents of /etc/modprobe.*
  job passed   : Collect information about kernel modules
  job passed   : Collect information about dpkg version
  job passed   : Provide links to requirements documents
  job passed   : Attach dump of udev database
  job passed   : Collect information about installed system (lsb-release)
  job passed   : Collect information about the EFI configuration
  job passed   : Attach information about block devices and their mount points.
  job passed   : Attaches JSON dumps of udev resource information.
job cannot be started: Attaches JSON dumps of system info tools
  job passed   : Collect information about installed software packages
  job passed   : Collect information about the running kernel
  job passed   : Collect information about hardware devices (udev)
  job passed   : Collect information about installation media (casper)
  job passed   : Attach a copy of /proc/cmdline
  job passed   : Collect information about installed snap packages
job cannot be started: Check that data for a complete result are present
  job passed   : System boot-up performance statistics
  job passed   : Print the tree of the time-critical chain of SystemD
  job passed   : Detect the presence of a Hardware Watchdog
  job passed   : Check if the hardware watchdog is properly configured
  job passed   : Test that the watchdog module can trigger a system reset
  job passed   : Post watchdog reset service check
1.25MB [00:02, 614kB/s, file=/home/baconyao/.local/share/checkbox-ng/submission_2024-11-14T00.36.28.994048.html]                                  
Submission url (/home/baconyao/.local/share/checkbox-ng/submission_2024-11-14T00.36.28.994048.c3_url.log) saved also to: file:///home/baconyao/.local/share/checkbox-ng/submission_2024-11-14T00.36.28.994048.html
32.0kB [00:00, 474kB/s, file=/home/baconyao/.local/share/checkbox-ng/submission_2024-11-14T00.36.28.994048.junit.xml]                             
Submission url (/home/baconyao/.local/share/checkbox-ng/submission_2024-11-14T00.36.28.994048.c3_url.log) saved also to: file:///home/baconyao/.local/share/checkbox-ng/submission_2024-11-14T00.36.28.994048.junit.xml
Problem with a '2_tar_file' report using 'com.canonical.plainbox::tar' exporter sent to '/home/baconyao/.local/share/checkbox-ng/submission_2024-11-14T00.36.28.994048.tar.xz' transport. Reason 'dict object' has no attribute 'com.canonical.certification::firmware/fwts_desktop_diagnosis'

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 359, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 837, in _handle_call
    return obj(*args, **dict(kwargs))
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/session/remote_assistant.py", line 847, in exposed_cache_report
    exporter.dump_from_session_manager(self._sa._manager, exported_stream)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/exporter/tar.py", line 74, in dump_from_session_manager
    exporter.dump_from_session_manager(manager, _s)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 205, in dump_from_session_manager
    self.dump(data, stream)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 175, in dump
    self.template.stream(data).dump(stream, encoding="utf-8")
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1612, in dump
    fp.writelines(iterable)
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3.10/tempfile.py", line 955, in writelines
    rv = file.writelines(iterable)
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1607, in <genexpr>
    iterable = (x.encode(encoding, errors) for x in self)  # type: ignore
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1656, in __next__
    return self._next()  # type: ignore
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1348, in generate
    yield self.environment.handle_exception()
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/providers/exporters/data/checkbox.json", line 115, in top-level template code
    "name": "{{ job_state_map[job_id].job.tr_summary() }}",
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 474, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'com.canonical.certification::firmware/fwts_desktop_diagnosis'

Traceback (most recent call last):
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/stages.py", line 686, in _export_results
    result = self._export_fn(exporter_id, transport)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/controller.py", line 829, in local_export
    rf = self.sa.cache_report(exporter_id, options)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/netref.py", line 240, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/netref.py", line 63, in syncreq
    return conn.sync_request(handler, proxy, *args)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 718, in sync_request
    return _async_res.value
  File "/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/async_.py", line 108, in value
    raise self._obj
plainbox.vendor.rpyc.core.vinegar/jinja2.exceptions._get_exception_class.<locals>.Derived: 'dict object' has no attribute 'com.canonical.certification::firmware/fwts_desktop_diagnosis'

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 359, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 837, in _handle_call
    return obj(*args, **dict(kwargs))
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/session/remote_assistant.py", line 847, in exposed_cache_report
    exporter.dump_from_session_manager(self._sa._manager, exported_stream)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/exporter/tar.py", line 74, in dump_from_session_manager
    exporter.dump_from_session_manager(manager, _s)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 205, in dump_from_session_manager
    self.dump(data, stream)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/exporter/jinja2.py", line 175, in dump
    self.template.stream(data).dump(stream, encoding="utf-8")
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1612, in dump
    fp.writelines(iterable)
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3.10/tempfile.py", line 955, in writelines
    rv = file.writelines(iterable)
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1607, in <genexpr>
    iterable = (x.encode(encoding, errors) for x in self)  # type: ignore
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1656, in __next__
    return self._next()  # type: ignore
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 1348, in generate
    yield self.environment.handle_exception()
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/snap/checkbox/10242/checkbox-runtime/lib/python3.10/site-packages/plainbox/impl/providers/exporters/data/checkbox.json", line 115, in top-level template code
    "name": "{{ job_state_map[job_id].job.tr_summary() }}",
  File "/snap/checkbox/10242/checkbox-runtime/usr/lib/python3/dist-packages/jinja2/environment.py", line 474, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'com.canonical.certification::firmware/fwts_desktop_diagnosis'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/checkbox22/current/bin/checkbox-cli", line 8, in <module>
    sys.exit(main())
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/checkbox_cli.py", line 166, in main
    return subcmd.invoked(ctx)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/controller.py", line 175, in invoked
    return self.connect_and_run(ctx.args.host, port)
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/controller.py", line 303, in connect_and_run
    keep_running = {
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/controller.py", line 464, in resume_or_start_new_session
    self.run_jobs()
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/controller.py", line 776, in run_jobs
    self.finish_session()
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/controller.py", line 702, in finish_session
    self._export_results()
  File "/snap/checkbox22/current/lib/python3.10/site-packages/checkbox_ng/launcher/stages.py", line 750, in _export_results
    traceback.print_tb(exc)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 53, in print_tb
    print_list(extract_tb(tb, limit=limit), file=file)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 72, in extract_tb
    return StackSummary.extract(walk_tb(tb), limit=limit)
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 364, in extract
    for f, lineno in frame_gen:
  File "/snap/checkbox22/current/usr/lib/python3.10/traceback.py", line 329, in walk_tb
    yield tb.tb_frame, tb.tb_lineno
AttributeError: 'jinja2.exceptions.UndefinedError' object has no attribute 'tb_frame'
pieqq commented 1 week ago

How to reproduce

I finally managed to reproduce the issue quite reliably, using the following steps:

  1. Setup a 22.04 LXC container that will run the Checkbox agent
  2. On both the host (Checkbox controller) and the LXC container (Checkbox agent), install Checkbox development version using steps in Contrib guide, making sure you install the certification-client provider as well
  3. On the agent, run checkbox-cli run-agent
  4. On the controller, prepare a launcher (mylauncher) that looks like this:
[test plan]
unit = com.canonical.certification::client-cert-iot-ubuntucore-22-automated
forced = yes

[agent]
normal_user = root

and use it: checkbox-cli control <IP of the LXC agent> mylauncher

  1. Deselect everything, except the Power Management / Warm Boot job
  2. When rerun screen appears, press F to finish the session and generate the submissions.

Investigation

The part of the JSON template that renders the rejected jobs is this:

    {%- for job_id in state.metadata.rejected_jobs %}
        {
            "id": "{{ job_id|strip_ns }}",
            "full_id": "{{ job_id }}",
            "name": "{{ job_state_map[job_id].job.tr_summary() }}",
            "plugin": "{{ job_state_map[job_id].job.plugin }}",
            "certification_status": "{{ job_state_map[job_id].effective_certification_status }}",
            "category": "{{ category_map[job_state_map[job_id].effective_category_id] }}",
            "category_id": "{{ job_state_map[job_id].effective_category_id }}",
            "template_id": {{ job_state_map[job_id].job.template_id | jsonify | safe }}
        }{%- if not loop.last -%},{%- endif %}
    {%- endfor %}

The problem is that at that stage (after a reboot), job_state_map only contains information about the jobs that have been selected for execution. Trying to get info for a job that has been rejected doesn't work.

Two issues there:

  1. Since this seems to work fine if the session is not resumed, it means something is lost at session resume time for the job_state_map
  2. The list of rejected jobs (state.metadata.rejected_jobs) should include more than just the job_id, but basically everything we need to expose at the end; this way, we don't need to depend on the job_state_map.