search

scylladb / scylla-manager

The Scylla Manager
https://manager.docs.scylladb.com/stable/
Other
48 stars 33 forks source link

Investigate GCE backup test failures #3863

Closed mikliapko closed 1 month ago

mikliapko commented 1 month ago

The GCE backup test fails by timeout (7 hours) in both master and manager-3.2 branches: https://jenkins.scylladb.com/view/scylla-manager/job/manager-3.2/job/sct-feature-test-backup-gce/28/ https://jenkins.scylladb.com/view/scylla-manager/job/manager-master/job/sct-feature-test-backup-gce/682/

mikliapko commented 1 month ago

Investigation result:

The place where test hangs forever:

< t:2024-05-10 21:08:25,964 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > + sudo apt-get install google-cloud-cli
< t:2024-05-10 21:08:26,191 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > Building dependency tree...
< t:2024-05-10 21:08:26,191 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > Reading state information...
< t:2024-05-10 21:08:26,314 f:db_log_reader.py l:114  c:sdcm.db_log_reader   p:DEBUG > 2024-05-10T21:08:26+00:00 manager-regression-manager--db-node-814781b7-0-1     !INFO | sudo[13288]: pam_unix(sudo:session): session closed for user root
< t:2024-05-10 21:08:26,315 f:db_log_reader.py l:114  c:sdcm.db_log_reader   p:DEBUG > 2024-05-10T21:08:26+00:00 manager-regression-manager--db-node-814781b7-0-1   !NOTICE | sudo[12979]: scylla-test : TTY=unknown ; PWD=/home/scylla-test ; USER=root ; COMMAND=/usr/bin/apt-get install google-cloud-cli
< t:2024-05-10 21:08:26,315 f:db_log_reader.py l:114  c:sdcm.db_log_reader   p:DEBUG > 2024-05-10T21:08:26+00:00 manager-regression-manager--db-node-814781b7-0-1     !INFO | sudo[12979]: pam_unix(sudo:session): session opened for user root by (uid=0)
< t:2024-05-10 21:08:26,394 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > The following packages were automatically installed and are no longer required:
< t:2024-05-10 21:08:26,395 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >   libestr0 libfastjson4
< t:2024-05-10 21:08:26,395 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > Use 'sudo apt autoremove' to remove them.
< t:2024-05-10 21:08:26,395 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > The following additional packages will be installed:
< t:2024-05-10 21:08:26,396 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >   google-cloud-cli-anthoscli
< t:2024-05-10 21:08:26,397 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > Suggested packages:
< t:2024-05-10 21:08:26,397 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >   google-cloud-cli-app-engine-java google-cloud-cli-app-engine-python
< t:2024-05-10 21:08:26,397 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >   google-cloud-cli-pubsub-emulator google-cloud-cli-bigtable-emulator
< t:2024-05-10 21:08:26,397 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >   google-cloud-cli-datastore-emulator kubectl
< t:2024-05-10 21:08:26,419 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > The following NEW packages will be installed:
< t:2024-05-10 21:08:26,420 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >   google-cloud-cli google-cloud-cli-anthoscli
< t:2024-05-10 21:08:26,426 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > 0 upgraded, 2 newly installed, 0 to remove and 37 not upgraded.
< t:2024-05-10 21:08:26,427 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > Need to get 106 MB of archives.
< t:2024-05-10 21:08:26,427 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > After this operation, 610 MB of additional disk space will be used.

google-cloud-cli pkg started to require additional package to be installed together google-cloud-cli-anthoscli. The apt install command should come with -y option.