freedomofpress / securedrop-workstation-ci

Repository for holding scripts related to running CI for SecureDrop Workstation (hosted on a Qubes machine)
GNU Affero General Public License v3.0
1 stars 0 forks source link

Nightly update process is failing since ~Sept 21 #77

Open legoktm opened 1 month ago

legoktm commented 1 month ago

Since Sept. 21, the nighly update process has been consistently failing:

Sep 21 04:30:01 ws-ci-bastion CRON[539585]: (wscirunner) CMD (/home/wscirunner/venv/bin/python /home/wscirunner/securedrop-workstation-ci/run.py --version 4.2 --update --save)
Sep 21 04:30:02 ws-ci-bastion ws-ci-runner[539586]: First reverting Qubes_4.2_D to snapshot update_20240920043551
Sep 21 04:30:02 ws-ci-bastion ws-ci-runner[539586]: Powering on Qubes_4.2_D
Sep 21 04:30:02 ws-ci-bastion ws-ci-runner[539586]: VM Qubes_4.2_D is not yet fully booted, waiting for it to be ready
...
Sep 21 04:31:23 ws-ci-bastion ws-ci-runner[539586]: VM Qubes_4.2_D is now ready, moving on with next steps
Sep 21 04:31:23 ws-ci-bastion ws-ci-runner[539586]: Applying updates on Qubes_4.2_D
Sep 21 04:31:23 ws-ci-bastion ws-ci-runner[539586]: Program running, PID is 3595
...
Sep 21 04:50:02 ws-ci-bastion ws-ci-runner[539586]: Program running, PID is 3595
Sep 21 04:50:07 ws-ci-bastion ws-ci-runner[539586]: ERROR: Program 3595 completed with Failure
Sep 21 04:50:07 ws-ci-bastion ws-ci-runner[539586]: ERROR: More info on process
Sep 21 04:50:08 ws-ci-bastion ws-ci-runner[539586]: (vim.vm.guest.ProcessManager.ProcessInfo) [
                                                       (vim.vm.guest.ProcessManager.ProcessInfo) {
                                                          dynamicType = <unset>,
                                                          dynamicProperty = (vmodl.DynamicProperty) [],
                                                          name = 'sudo',
                                                          pid = 3595,
                                                          owner = 'user',
                                                          cmdLine = '"/usr/bin/sudo" /usr/bin/qubesctl --show-output state.sls update.qubes-dom0',
                                                          startTime = 2024-09-21T04:31:29Z,
                                                          endTime = 2024-09-21T04:50:09Z,
                                                          exitCode = 1
                                                       }
                                                    ]
legoktm commented 1 month ago

(Booting up the VM manually to see what is going wrong...)

legoktm commented 1 month ago

Unsatisfyingly, the two commands worked properly when I booted up Qubes_4.2_D and ran it in a dom0 terminal.