search

GoogleCloudDataproc / initialization-actions

Run in all nodes of your cluster before the cluster starts - lets you customize your cluster
https://cloud.google.com/dataproc/init-actions
Apache License 2.0
588 stars 512 forks source link

[gpu] apt-get update Init script seeing broken repositories #1129

Open jmcculloch-google opened 10 months ago

jmcculloch-google commented 10 months ago

Creating a new Dataproc cluster using the install_gpu_driver.sh init script results in the following two issues:

  1. An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.mysql.com/apt/debian buster InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B7B3B788A8D3785C

=> Current workaround is to remove the MySQL repository from being updated via: mv /etc/apt/sources.list.d/mysql.list /tmp/mysql.list

  1. E: Repository 'https://packages.cloud.google.com/apt google-cloud-monitoring-buster-all InRelease' changed its 'Origin' value from 'google-cloud-monitoring-buster' to 'namespaces/cloud-ops-agents-artifacts/repositories/google-cloud-monitoring-buster-all' E: Repository 'https://packages.cloud.google.com/apt google-cloud-monitoring-buster-all InRelease' changed its 'Label' value from 'google-cloud-monitoring-buster' to 'namespaces/cloud-ops-agents-artifacts/repositories/google-cloud-monitoring-buster-all'

=> Workaround is to allow for the repository change: apt-get --allow-releaseinfo-change update

Full log:

+ execute_with_retries 'apt-get update'
+ local -r 'cmd=apt-get update'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'apt-get update'
++ apt-get update
Get:1 https://storage.googleapis.com/goog-dataproc-bigtop-repo-europe-west4/2_0_deb10_20231206_122000-RC01 dataproc InRelease [3708 B]
Hit:2 http://cloud.r-project.org/bin/linux/debian buster-cran40/ InRelease
Get:3 https://download.docker.com/linux/debian buster InRelease [53.9 kB]
Get:4 https://repo.mysql.com/apt/debian buster InRelease [22.1 kB]
Hit:5 https://storage.googleapis.com/dataproc-bigtop-repo/2_0_deb10_20231206_122000-RC01 dataproc InRelease
Get:6 https://packages.cloud.google.com/apt google-cloud-logging-buster-all InRelease [5104 B]
Hit:7 https://deb.debian.org/debian buster InRelease
Get:8 https://deb.debian.org/debian-security buster/updates InRelease [34.8 kB]
Get:9 https://storage.googleapis.com/goog-dataproc-bigtop-repo-europe-west4/2_0_deb10_20231206_122000-RC01 dataproc/contrib Sources [9547 B]
Get:10 https://packages.cloud.google.com/apt google-cloud-monitoring-buster-all InRelease [1237 B]
Hit:11 https://deb.debian.org/debian buster-updates InRelease
Get:12 https://storage.googleapis.com/goog-dataproc-bigtop-repo-europe-west4/2_0_deb10_20231206_122000-RC01 dataproc/contrib amd64 Packages [21.4 kB]
Get:13 https://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease [5136 B]
Get:14 https://deb.debian.org/debian buster-backports InRelease [51.4 kB]
Get:15 https://packages.cloud.google.com/apt cloud-sdk-buster InRelease [6396 B]
Get:16 https://packages.adoptium.net/artifactory/deb buster InRelease [6634 B]
Get:17 https://download.docker.com/linux/debian buster/stable amd64 Packages [43.3 kB]
Err:4 https://repo.mysql.com/apt/debian buster InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B7B3B788A8D3785C
Get:18 https://deb.debian.org/debian-security buster/updates/main Sources [367 kB]
Get:19 https://deb.debian.org/debian-security buster/updates/main amd64 Packages [580 kB]
Get:20 https://deb.debian.org/debian-security buster/updates/main Translation-en [314 kB]
Get:21 https://packages.cloud.google.com/apt google-compute-engine-buster-stable/main amd64 Packages [2281 B]
Get:22 https://packages.cloud.google.com/apt cloud-sdk-buster/main amd64 Packages [572 kB]
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.mysql.com/apt/debian buster InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B7B3B788A8D3785C
E: Repository 'https://packages.cloud.google.com/apt google-cloud-monitoring-buster-all InRelease' changed its 'Origin' value from 'google-cloud-monitoring-buster' to 'namespaces/cloud-ops-agents-artifacts/repositories/google-cloud-monitoring-buster-all'
E: Repository 'https://packages.cloud.google.com/apt google-cloud-monitoring-buster-all InRelease' changed its 'Label' value from 'google-cloud-monitoring-buster' to 'namespaces/cloud-ops-agents-artifacts/repositories/google-cloud-monitoring-buster-all'
cjac commented 10 months ago

Thank you! I'll get a PR opened shortly.

jayadeep-jayaraman commented 10 months ago

Please use the latest 2.0 image if you are using an older version of 2.0 image in your dataproc cluster creation script. There was a change in public key of mysql repo in Dec'2023 because of which you might be facing the issue.

With a recent image I am able to see it working fine

jjayadeep_google_com@jj-deb-20-m:~$ sudo apt-get update
Get:1 https://packages.cloud.google.com/apt google-cloud-logging-buster-all InRelease [5104 B]
Hit:2 http://cloud.r-project.org/bin/linux/debian buster-cran40/ InRelease                                                                                                                                                                                                             
Get:3 https://packages.cloud.google.com/apt google-cloud-monitoring-buster-all InRelease [5092 B]                                                                                                                                                     
Get:4 https://storage.googleapis.com/goog-dataproc-bigtop-repo-us-central1/2_0_deb10_20240111_122000-RC01 dataproc InRelease [3708 B]                                                                                                                 
Get:5 https://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease [5136 B]                                                                                                     
Get:6 https://download.docker.com/linux/debian buster InRelease [53.9 kB]                                                                                                       
Get:7 https://packages.cloud.google.com/apt cloud-sdk-buster InRelease [6396 B]                                                                                                   
Hit:8 https://repo.mysql.com/apt/debian buster InRelease                                                                                                                                    
Hit:9 https://storage.googleapis.com/dataproc-bigtop-repo/2_0_deb10_20240111_122000-RC01 dataproc InRelease                                                 
Get:10 https://packages.adoptium.net/artifactory/deb buster InRelease [6634 B]