roachtest: c2c/kv0 failed [apt flake in require.NoError not detected]

[cockroach-teamcity]

roachtest.c2c/kv0 failed with artifacts on release-24.1 @ cdc755b976cc21251589de157e8180189c061f1b:

                      |   | echo "Downloading https://dl.grafana.com/enterprise/release/grafana-enterprise_9.2.3_amd64.deb" &&
                      |   | curl https://dl.grafana.com/enterprise/release/grafana-enterprise_9.2.3_amd64.deb -sS -o grafana-enterprise_9.2.3_amd64.deb &&
                      |   | sudo dpkg -i grafana-enterprise_9.2.3_amd64.deb &&
                      |   | sudo mkdir -p /var/lib/grafana/dashboards
                      |   | ``````
                      |   | stdout: <empty>
                      |   | stderr:E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 4146 (unattended-upgr)
                      |   | E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
                      | Wraps: (2) TRANSIENT_ERROR(apt_problem)
                      | Wraps: (3) exit status 100
                      | Error types: (1) *hintdetail.withDetail (2) errors.TransientError (3) *exec.ExitError
                    Wraps: (5) secondary error attachment
                      | TRANSIENT_ERROR(apt_problem): exit status 100
                      | (1) Node 7. Command with error:
                      |   | ``````
                      |   |
                      |   | sudo apt-get install -qqy apt-transport-https &&
                      |   | sudo apt-get install -qqy software-properties-common &&
                      |   | sudo apt-get install -y adduser libfontconfig1 &&
                      |   | echo "Downloading https://dl.grafana.com/enterprise/release/grafana-enterprise_9.2.3_amd64.deb" &&
                      |   | curl https://dl.grafana.com/enterprise/release/grafana-enterprise_9.2.3_amd64.deb -sS -o grafana-enterprise_9.2.3_amd64.deb &&
                      |   | sudo dpkg -i grafana-enterprise_9.2.3_amd64.deb &&
                      |   | sudo mkdir -p /var/lib/grafana/dashboards
                      |   | ``````
                      |   | stdout: <empty>
                      |   | stderr:E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 4146 (unattended-upgr)
                      |   | E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
                      | Wraps: (2) TRANSIENT_ERROR(apt_problem)
                      | Wraps: (3) exit status 100
                      | Error types: (1) *hintdetail.withDetail (2) errors.TransientError (3) *exec.ExitError
                    Wraps: (6) Node 7. Command with error:
                      | ``````
                      |
                      | sudo apt-get install -qqy apt-transport-https &&
                      | sudo apt-get install -qqy software-properties-common &&
                      | sudo apt-get install -y adduser libfontconfig1 &&
                      | echo "Downloading https://dl.grafana.com/enterprise/release/grafana-enterprise_9.2.3_amd64.deb" &&
                      | curl https://dl.grafana.com/enterprise/release/grafana-enterprise_9.2.3_amd64.deb -sS -o grafana-enterprise_9.2.3_amd64.deb &&
                      | sudo dpkg -i grafana-enterprise_9.2.3_amd64.deb &&
                      | sudo mkdir -p /var/lib/grafana/dashboards
                      | ``````
                      | stdout: <empty>
                      | stderr:E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 4146 (unattended-upgr)
                      | E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
                    Wraps: (7) TRANSIENT_ERROR(apt_problem)
                    Wraps: (8) exit status 100
                    Error types: (1) *markers.withMark (2) *withstack.withStack (3) *errutil.withPrefix (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *hintdetail.withDetail (7) errors.TransientError (8) *exec.ExitError
    Test:           c2c/kv0
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/c2c/kv0/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-39548

[blathers-crl[bot]]

cc @cockroachdb/test-eng

[msbutler]

grafana failed to start. routing to test eng

[renatolabs]

Hrm, pretty sure the apt flake was not automatically triaged because of the use of require.NoError. We will need to check if there's a way observe roachprod's transient error from the error the testify library generates.