search

scylladb / scylla-manager

The Scylla Manager
https://manager.docs.scylladb.com/stable/
Other
51 stars 33 forks source link

Unable to locate package scylla-manager-server, Unable to locate package scylla-manager-client errors during node setup #3988

Closed temichus closed 4 days ago

temichus commented 3 weeks ago

SCT failed at the node setup stage with the error sdcm.cluster.NodeSetupFailed on latest enterprise image scylla-enterprise-2024.3.0-dev-x86_64-2024-08-21T01-55-20 | ami-0c6e012613f908379

JOB: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/Serverless_v2/job/perf-regression-elasticity-test-serverless-v2-low-load/18/execution/node/182/log/

message 

 10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Command: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=120 -o Dpkg::Options::="--force-confold" -o Dpkg::Options::="--force-confdef" install -y scylla-manager-server scylla-manager-client'
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Exit code: 100
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Stdout:
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Reading package lists...
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Building dependency tree...
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Reading state information...
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Stderr:
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > E: Unable to locate package scylla-manager-server
10:42:15  < t:2024-08-21 08:42:15,414 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > E: Unable to locate package scylla-manager-client

The issue also reproduced on images that worked previously (scylla-enterprise-2024.3.0-dev-x86_64-2024-08-19T01-44-22 | ami-0703eb70ef9806b2f)

Node setup 21 Aug - failed: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/Serverless_v2/job/perf-regression-elasticity-test-serverless-v2/15/execution/node/182/log/

Node setup 19 Aug - passed: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/Serverless_v2/job/perf-regression-elasticity-test-serverless-v2/11/execution/node/182/log/?consoleFull

temichus commented 3 weeks ago

cc @mykaul

mykaul commented 3 weeks ago

@Annamikhlin , @karol-kokoszka - please look into it. Either some repo issue - or the new gpg key issue?

karol-kokoszka commented 3 weeks ago

It seem to be connected to the new gpg key. I added manager-3.3 to my local apt, and cannot update because of:

Err:12 https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3 stable InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A43E06657BAC99E3

@Annamikhlin the .list file looks like that:

deb [arch=amd64,arm64 signed-by=/etc/apt/keyrings/scylladb.gpg] https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3 stable main

How can I import the new key ?

Things like:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A43E06657BAC99E3

or

wget -qO - https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3/A43E06657BAC99E3.gpg.key | sudo apt-key add -

doesn't work.

Annamikhlin commented 3 weeks ago

@karol-kokoszka - It was imported for me locally

# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A43E06657BAC99E3
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
Executing: /tmp/apt-key-gpghome.jYy3yLzlxh/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys A43E06657BAC99E3
gpg: key A43E06657BAC99E3: public key "ScyllaDB Package Signing Key 2024 (RSA) <security@scylladb.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Node setup 21 Aug - failed: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/Serverless_v2/job/perf-regression-elasticity-test-serverless-v2/15/execution/node/182/log/

@temichus - to which byo job number your test is refer to?

byo_job_path
/scylla-master/byo/byo_build_tests_dtest
karol-kokoszka commented 3 weeks ago

@Annamikhlin try 

sudo apt-get update

Err:13 https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3 stable InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A43E06657BAC99E3

mykaul commented 3 weeks ago

@karol-kokoszka - are you failing to update the key first? or did you update and it still doesn't work?

karol-kokoszka commented 3 weeks ago

I updated the key, I see the same output as @Annamikhlin provided, but the sudo apt-get update still fails.

➜  _review sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A43E06657BAC99E3
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
Executing: /tmp/apt-key-gpghome.B4n3V9tabw/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys A43E06657BAC99E3
gpg: key A43E06657BAC99E3: public key "ScyllaDB Package Signing Key 2024 (RSA) <security@scylladb.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

➜  _review sudo apt-get update                                                                                                                                                   
Hit:1 http://pl.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://dl.google.com/linux/chrome/deb stable InRelease                                                                                                                                               
Get:3 http://pl.archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]                                                                                                                                 
Hit:4 http://pl.archive.ubuntu.com/ubuntu jammy-backports InRelease                                                                                                                                        
Get:5 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]                                                                                                                                  
Hit:6 https://repo.steampowered.com/steam stable InRelease                                                                                                                                                 
Hit:7 https://pkg.cloudflareclient.com jammy InRelease                                                                                                                                                     
Ign:8 https://repo.skype.com/deb stable InRelease                                                                                                                                                          
Hit:9 https://apt.releases.hashicorp.com jammy InRelease                                                                                                                                                   
Hit:10 https://download.docker.com/linux/ubuntu jammy InRelease                                                                                                                                            
Get:11 https://deb.nodesource.com/node_16.x jammy InRelease [4 583 B]                                                                                                                                      
Get:12 https://cli.github.com/packages stable InRelease [3 917 B]                                                                                                                            
Get:13 https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3 stable InRelease [7 550 B]                                 
Err:13 https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3 stable InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A43E06657BAC99E3
Hit:14 https://ppa.launchpadcontent.net/linuxuprising/java/ubuntu jammy InRelease
Hit:15 https://ppa.launchpadcontent.net/obsproject/obs-studio/ubuntu jammy InRelease
Ign:8 https://repo.skype.com/deb stable InRelease
Ign:8 https://repo.skype.com/deb stable InRelease
Err:8 https://repo.skype.com/deb stable InRelease                                                                                                                                                          
  503  Service Unavailable [IP: 104.84.60.178 443]
Reading package lists... Done                                                                                                                                                                              
N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'https://pkg.cloudflareclient.com jammy InRelease' doesn't support architecture 'i386'
W: GPG error: https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3 stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A43E06657BAC99E3
E: The repository 'https://downloads.scylladb.com/downloads/scylla-manager/deb/ubuntu/scylladb-manager-3.3 stable InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
temichus commented 3 weeks ago

@karol-kokoszka - It was imported for me locally

# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A43E06657BAC99E3
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
Executing: /tmp/apt-key-gpghome.jYy3yLzlxh/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys A43E06657BAC99E3
gpg: key A43E06657BAC99E3: public key "ScyllaDB Package Signing Key 2024 (RSA) <security@scylladb.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Node setup 21 Aug - failed: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/Serverless_v2/job/perf-regression-elasticity-test-serverless-v2/15/execution/node/182/log/

@temichus - to which byo job number your test is refer to?

byo_job_path
/scylla-master/byo/byo_build_tests_dtest

Job has just "scylla_version or scylla_ami_id" parameters and did non use "byo" functionality

ami-0703eb70ef9806b2f was used in 19 Aug runs when I ran job with scylla_version: master:latest parameter(when node setup was passed)

Annamikhlin commented 3 weeks ago

ok, reproduced it locally and found an issue with configuration of ListUrl in pkg https://github.com/scylladb/scylla-pkg/blob/3784ef0b4520da8ddbd79432587ee6237c4740d4/scripts/jenkins-pipelines/manager/branch-specific.properties#L24 pointed to master instead of branch-3.3

karol-kokoszka commented 3 weeks ago

@Annamikhlin will we need to promote Scylla-Manager once again ?

Annamikhlin commented 3 weeks ago

@Annamikhlin will we need to promote Scylla-Manager once again ?

Yes

Annamikhlin commented 3 weeks ago

https://jenkins.scylladb.com/view/scylla-manager/job/manager-3.3/job/manager-build/48/ - manager-build passed with a fix commit

karol-kokoszka commented 3 weeks ago

So, now it's manager-release and manager-promote to run ?

Annamikhlin commented 3 weeks ago

So, now it's manager-release and manager-promote to run ?

manager-promote is currently running https://jenkins.scylladb.com/view/scylla-manager/job/manager-3.3/job/manager-promote/4/ create repo already passed. checking locally

karol-kokoszka commented 3 weeks ago

Thank you !

Annamikhlin commented 3 weeks ago

The following steps worked for me:

mkdir -p /etc/apt/keyrings
gpg --homedir /tmp --no-default-keyring --keyring /etc/apt/keyrings/scylladb.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys A43E06657BAC99E3
wget -O /etc/apt/sources.list.d/scylla-manager.list https://downloads.scylladb.com/deb/ubuntu/scylladb-manager-3.3.list
apt-get update
apt-get install scylla-manager-server scylla-manager-client
Annamikhlin commented 3 weeks ago

@tzach - need to update the site with version manager-3.3 and new gpg key A43E06657BAC99E3 for 3.3.1 the latest version on the site is manager-3.2 - https://www.scylladb.com/download/?platform=ubuntu-22.04&version=manager-3.2#manager

karol-kokoszka commented 3 weeks ago

I confirm that it works for me as well.

mykaul commented 3 weeks ago

@mikliapko , @rayakurl - we need to add post release tests, to ensure we catch such issues faster. Could be an 'artifacts' tests post release.

rayakurl commented 3 weeks ago

@mikliapko , @rayakurl - we need to add post release tests, to ensure we catch such issues faster. Could be an 'artifacts' tests post release.

@mikliapko thanks for opening the issue, we can discuss the priority after the restore effort is over.

aleksbykov commented 2 weeks ago

Issue still reproduced with job: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/Serverless_v2/job/perf-regression-elasticity-test-serverless-v2-hight-load/

yaronkaikov commented 2 weeks ago

Issue still reproduced with job: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/Serverless_v2/job/perf-regression-elasticity-test-serverless-v2-hight-load/

Please verify that this branch you are working on is rebased, it should have https://github.com/scylladb/scylla-cluster-tests/pull/8173/commits/7b9a46e3dbac2a9f104a732ae6b6a1649a4fe075 otherwise it will not work

tzach commented 2 weeks ago

@tzach - need to update the site with version manager-3.3 and new gpg key A43E06657BAC99E3 for 3.3.1 the latest version on the site is manager-3.2 - https://www.scylladb.com/download/?platform=ubuntu-22.04&version=manager-3.2#manager

Done. Please validate.

yaronkaikov commented 2 weeks ago

@tzach - need to update the site with version manager-3.3 and new gpg key A43E06657BAC99E3 for 3.3.1 the latest version on the site is manager-3.2 - https://www.scylladb.com/download/?platform=ubuntu-22.04&version=manager-3.2#manager

Done. Please validate.

Looks good :-)

mikliapko commented 2 weeks ago

AFAIU we faced this issue because of the problem in promote job, right? In current approach we trigger tests in build job and use the link to .repo/.list file from unstable. That's why we missed this problem. I wonder if we can extend promote pipeline to trigger simple test that installs Manager from official source (https://downloads.scylladb.com/deb/ubuntu/scylladb-manager-3.3.list, for example) after promotion stage?

@yaronkaikov @Annamikhlin What do you think?

yaronkaikov commented 2 weeks ago

AFAIU we faced this issue because of the problem in promote job, right? In current approach we trigger tests in build job and use the link to .repo/.list file from unstable. That's why we missed this problem. I wonder if we can extend promote pipeline to trigger simple test that installs Manager from official source (https://downloads.scylladb.com/deb/ubuntu/scylladb-manager-3.3.list, for example) after promotion stage?

@yaronkaikov @Annamikhlin What do you think?

sure we can

I suggest to open an issue in scylla-pkg with details about the test you wish to trigger

mikliapko commented 1 week ago

AFAIU we faced this issue because of the problem in promote job, right? In current approach we trigger tests in build job and use the link to .repo/.list file from unstable. That's why we missed this problem. I wonder if we can extend promote pipeline to trigger simple test that installs Manager from official source (https://downloads.scylladb.com/deb/ubuntu/scylladb-manager-3.3.list, for example) after promotion stage? @yaronkaikov @Annamikhlin What do you think?

sure we can

I suggest to open an issue in scylla-pkg with details about the test you wish to trigger

Issued the PR https://github.com/scylladb/scylla-pkg/pull/4302

mykaul commented 4 days ago

I think this issue is solved. What's left here?

yaronkaikov commented 4 days ago

I think this issue is solved. What's left here?

Only merging https://github.com/scylladb/scylla-pkg/pull/4302 Probably will be done today

yaronkaikov commented 4 days ago

I think this issue is solved. What's left here?

Only merging scylladb/scylla-pkg#4302 Probably will be done today

Once merge this issue will be closed