search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
57 stars 95 forks source link

Test stuck upon 'gpg --homedir /tmp --no-default-keyring --keyring /etc/apt/keyrings/scylladb.gpg' #9017

Open soyacz opened 1 month ago

soyacz commented 1 month ago

As in the title. Test details:

Packages

Base Scylla version: 6.2.0~rc2-20241002.93700ff5d1ce with build-id 57e6a907e0a03a185c4145bb8c7d6316b519b14c Target Scylla version (or git commit hash): 6.3.0~dev-20241018.b11d50f59191 with build-id d5fe38f8fd12d9b834688320151f36fb8c1e050d

Kernel Version: 6.8.0-1016-aws

Issue description

Describe your issue in detail and steps it took to produce it.

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 4 nodes (i4i.2xlarge)

Scylla Nodes used in this run:

OS / Image: ami-0f289f43805741bb7 (aws: undefined_region)

Test: rolling-upgrade-ami-test Test id: cf77d5a4-d5eb-4b39-99a3-3e0856f644b1 Test name: scylla-master/rolling-upgrade/rolling-upgrade-ami-test Test method: upgrade_test.UpgradeTest.test_rolling_upgrade Test config file(s):

Logs and commands - Restore Monitor Stack command: `$ hydra investigate show-monitor cf77d5a4-d5eb-4b39-99a3-3e0856f644b1` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=cf77d5a4-d5eb-4b39-99a3-3e0856f644b1) - Show all stored logs command: `$ hydra investigate show-logs cf77d5a4-d5eb-4b39-99a3-3e0856f644b1` ## Logs: - **db-cluster-cf77d5a4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/db-cluster-cf77d5a4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/db-cluster-cf77d5a4.tar.gz) - **sct-runner-events-cf77d5a4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/sct-runner-events-cf77d5a4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/sct-runner-events-cf77d5a4.tar.gz) - **sct-cf77d5a4.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/sct-cf77d5a4.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/sct-cf77d5a4.log.tar.gz) - **loader-set-cf77d5a4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/loader-set-cf77d5a4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/loader-set-cf77d5a4.tar.gz) - **monitor-set-cf77d5a4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/monitor-set-cf77d5a4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/cf77d5a4-d5eb-4b39-99a3-3e0856f644b1/20241019_092629/monitor-set-cf77d5a4.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/rolling-upgrade/job/rolling-upgrade-ami-test/246/) [Argus](https://argus.scylladb.com/test/cc80460f-2278-4fe5-8781-1776502bc7e1/runs?additionalRuns[]=cf77d5a4-d5eb-4b39-99a3-3e0856f644b1)
soyacz commented 1 month ago

@fruch possibly apt running in parallel locks it. Maybe we could use flock to wait for it?

flock -w 300 /var/lib/dpkg/lock-frontend gpg --homedir /tmp --no-default-keyring --keyring /etc/apt/keyrings/scylladb.gpg