search

canonical / cloud-init

Official upstream for the cloud-init: cloud instance initialization
https://cloud-init.io/
Other
3.02k stars 889 forks source link

cloud init 23.1.2 fails since this week on AWS, DigitalOcean and ExoScale #4117

Closed boaks closed 1 year ago

boaks commented 1 year ago

A cloud-init fails since this week. It has still been working last week.

cloud-config-dev.yaml

package_upgrade: true

packages:
# java - runtime for java application
 - openjdk-17-jre-headless
# fail2ban - network protection 
 - fail2ban

snap:
  commands:
   - snap refresh
# public x509 certificate / letsencrypt 
   - snap install --classic certbot

disable_root: false

users:
 - name: cali
   gecos: (Cf) Californium Demo Server
   lock_passwd: true

AWS, Ubuntu 20.04.6 LTS, cloud-init 23.1.2-0ubuntu0~20.04.1

finish: modules-final: FAIL: running modules for final

Same on DigitalOcean and ExoScale. Also on retry. Yesterday and today.

(If required, I may upload the "collect-logs", but I'm not sure, if that leaks credentials.)

blackboxsw commented 1 year ago

@boaks thanks for filing this bug and making cloud-init better. I'm marking this incomplete as I think we need more information here in order to determine the issue.

Can you please minimally provide the output of cloud-init status --long cloud-init --version

and any egrep -i 'Tracebacks|ERROR|WARNING' /var/log/cloud-init.log and any errors seen in /var/log/cloud-init-output.log We'll drop the 'incomplete' label once we receive a response on this issue.

boaks commented 1 year ago

The version was already provided, anyway here's what you requested:

sh> cloud-init --version

/usr/bin/cloud-init 23.1.2-0ubuntu0~20.04.1

sh> cloud-init status --long

status: error boot_status_code: enabled-by-generator last_update: Thu, 18 May 2023 05:43:15 +0000 detail: ('package-update-upgrade-install', ProcessExecutionError("Unexpected error while running command.\nCommand: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'openjdk-17-jre-headless', 'fail2ban']\nExit code: 100\nReason: -\nStdout: -\nStderr: -"))

sh> egrep -i 'Tracebacks|ERROR|WARNING' /var/log/cloud-init.log

2023-05-18 05:43:14,821 - util.py[WARNING]: Failed to install packages: ['openjdk-17-jre-headless', 'fail2ban'] cloudinit.subp.ProcessExecutionError: Unexpected error while running command. 2023-05-18 05:43:14,841 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one 2023-05-18 05:43:14,848 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed cloudinit.subp.ProcessExecutionError: Unexpected error while running command.

sh> egrep -i 'Tracebacks|ERROR|WARNING' /var/log/cloud-init-output.log

Warning: Stopping motd-news.service, but it can still be activated by: Warning: Stopping multipath-tools.service, but it can still be activated by: Warning from /etc/apparmor.d/sbin.dhclient (/etc/apparmor.d/sbin.dhclient line 4): apparmor_parser: File '/etc/apparmor.d/sbin.dhclient' missing feature abi, falling back to default policy feature abi update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. Exception in thread "main" java.lang.InternalError: Error loading java.security file dpkg: error processing package ca-certificates-java (--configure): installed ca-certificates-java package post-installation script subprocess returned error exit status 1 dpkg: error processing package openjdk-17-jre-headless:amd64 (--configure): No apport report written because the error message indicates its a followup error from a previous failure. Errors were encountered while processing: E: Sub-process /usr/bin/dpkg returned an error code (1) 2023-05-18 05:43:14,821 - util.py[WARNING]: Failed to install packages: ['openjdk-17-jre-headless', 'fail2ban'] 2023-05-18 05:43:14,841 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one 2023-05-18 05:43:14,848 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed

Runnning apt update and apt full-upgraderesults in:

Reading package lists... Done Building dependency tree
Reading state information... Done Calculating upgrade... Done The following packages were automatically installed and are no longer required: libfwupdplugin1 libxmlb1 Use 'apt autoremove' to remove them. Get more security updates through Ubuntu Pro with 'esm-apps' enabled: fail2ban Learn more about Ubuntu Pro on AWS at https://ubuntu.com/aws/pro 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2 not fully installed or removed. After this operation, 0 B of additional disk space will be used.

Setting up openjdk-17-jre-headless:amd64 (17.0.7+7~us1-0ubuntu1~20.04) ... update-alternatives: using /usr/lib/jvm/java-17-openjdk-amd64/bin/java to provide /usr/bin/java (java) in auto mode update-alternatives: using /usr/lib/jvm/java-17-openjdk-amd64/bin/jpackage to provide /usr/bin/jpackage (jpackage) in auto mode update-alternatives: using /usr/lib/jvm/java-17-openjdk-amd64/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode update-alternatives: using /usr/lib/jvm/java-17-openjdk-amd64/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode update-alternatives: using /usr/lib/jvm/java-17-openjdk-amd64/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode Setting up ca-certificates-java (20190405ubuntu1) ... head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory Adding debian:UCA_Global_G2_Root.pem Adding debian:Entrust_Root_Certification_Authority.pem Adding debian:Trustwave_Global_ECC_P256_Certification_Authority.pem Adding debian:Starfield_Services_Root_CertificateAuthority-_G2.pem Adding debian:DigiCert_Global_Root_G3.pem Adding debian:Hongkong_Post_Root_CA_1.pem Adding debian:Certum_Trusted_Root_CA.pem Adding debian:EC-ACC.pem Adding debian:GlobalSign_Root_CA.pem Adding debian:emSign_RootCA-_C1.pem Adding debian:Security_Communication_Root_CA.pem Adding debian:Secure_Global_CA.pem Adding debian:ISRG_Root_X1.pem Adding debian:Buypass_Class_2_Root_CA.pem Adding debian:Go_Daddy_Root_CertificateAuthority-_G2.pem Adding debian:GTS_Root_R1.pem Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2015.pem Adding debian:GlobalSign_Root_E46.pem Adding debian:OISTE_WISeKey_Global_Root_GB_CA.pem Adding debian:Microsec_e-Szigno_Root_CA_2009.pem Adding debian:T-TeleSec_GlobalRoot_Class_3.pem Adding debian:Entrust_Root_CertificationAuthority-_G2.pem Adding debian:NAVER_Global_Root_Certification_Authority.pem Adding debian:IdenTrust_Public_Sector_Root_CA_1.pem Adding debian:UCA_Extended_Validation_Root.pem Adding debian:AC_RAIZ_FNMT-RCM_SERVIDORES_SEGUROS.pem Adding debian:Certum_EC-384_CA.pem Adding debian:DigiCert_Assured_ID_Root_G3.pem Adding debian:Starfield_Root_CertificateAuthority-_G2.pem Adding debian:ePKI_Root_Certification_Authority.pem Adding debian:QuoVadis_Root_CA_1_G3.pem Adding debian:Certum_Trusted_Network_CA_2.pem Adding debian:Go_Daddy_Class_2_CA.pem Adding debian:Network_Solutions_Certificate_Authority.pem Adding debian:SSL.com_Root_Certification_Authority_RSA.pem Adding debian:Microsoft_RSA_Root_Certificate_Authority_2017.pem Adding debian:GTS_Root_R4.pem Adding debian:GDCA_TrustAUTH_R5_ROOT.pem Adding debian:D-TRUST_Root_Class_3_CA_2_EV_2009.pem Adding debian:GlobalSign_ECC_RootCA-_R5.pem Adding debian:OISTE_WISeKey_Global_Root_GC_CA.pem Adding debian:Cybertrust_Global_Root.pem Adding debian:Comodo_AAA_Services_root.pem Adding debian:SwissSign_GoldCA-_G2.pem Adding debian:Amazon_Root_CA_3.pem Adding debian:SecureTrust_CA.pem Adding debian:USERTrust_ECC_Certification_Authority.pem Adding debian:e-Szigno_Root_CA_2017.pem Adding debian:Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem Adding debian:Trustwave_Global_ECC_P384_Certification_Authority.pem Adding debian:TWCA_Root_Certification_Authority.pem Adding debian:ACCVRAIZ1.pem Adding debian:GTS_Root_R2.pem Adding debian:CFCA_EV_ROOT.pem Adding debian:CA_Disig_Root_R2.pem Adding debian:DigiCert_Global_Root_G2.pem Adding debian:T-TeleSec_GlobalRoot_Class_2.pem Adding debian:E-Tugra_Certification_Authority.pem Adding debian:Certigna_Root_CA.pem Adding debian:QuoVadis_Root_CA_3_G3.pem Adding debian:Amazon_Root_CA_2.pem Adding debian:emSign_ECC_RootCA-_G3.pem Adding debian:Entrust.net_Premium_2048_Secure_Server_CA.pem Adding debian:certSIGN_ROOT_CA.pem Adding debian:AffirmTrust_Commercial.pem Adding debian:COMODO_RSA_Certification_Authority.pem Adding debian:SSL.com_Root_Certification_Authority_ECC.pem Adding debian:GlobalSign_Root_R46.pem Adding debian:Atos_TrustedRoot_2011.pem Adding debian:Certum_Trusted_Network_CA.pem Adding debian:certSIGN_Root_CA_G2.pem Adding debian:GlobalSign_RootCA-_R2.pem Adding debian:TeliaSonera_Root_CA_v1.pem Adding debian:TWCA_Global_Root_CA.pem Adding debian:Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.pem Adding debian:DigiCert_Assured_ID_Root_CA.pem Adding debian:GlobalSign_ECC_RootCA-_R4.pem Adding debian:Amazon_Root_CA_1.pem Adding debian:GlobalSign_RootCA-_R6.pem Adding debian:GlobalSign_RootCA-_R3.pem Adding debian:QuoVadis_Root_CA_2.pem Adding debian:Trustwave_Global_Certification_Authority.pem Adding debian:NetLockArany=Class_Gold=_Főtanúsítvány.pem Adding debian:Izenpe.com.pem Adding debian:Microsoft_ECC_Root_Certificate_Authority_2017.pem Adding debian:COMODO_Certification_Authority.pem Adding debian:Staat_der_Nederlanden_EV_Root_CA.pem Adding debian:TUBITAK_Kamu_SM_SSL_KokSertifikasi-_Surum_1.pem Adding debian:COMODO_ECC_Certification_Authority.pem Adding debian:GTS_Root_R3.pem Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem Adding debian:Starfield_Class_2_CA.pem Adding debian:DigiCert_Trusted_Root_G4.pem Adding debian:D-TRUST_Root_Class_3_CA_2_2009.pem Adding debian:DigiCert_High_Assurance_EV_Root_CA.pem Adding debian:SecureSign_RootCA11.pem Adding debian:Entrust_Root_CertificationAuthority-_G4.pem Adding debian:SSL.com_EV_Root_Certification_Authority_ECC.pem Adding debian:ANF_Secure_Server_Root_CA.pem Adding debian:DigiCert_Global_Root_CA.pem Adding debian:IdenTrust_Commercial_Root_CA_1.pem Adding debian:Hongkong_Post_Root_CA_3.pem Adding debian:XRamp_Global_CA_Root.pem Adding debian:emSign_RootCA-_G1.pem Adding debian:Security_Communication_RootCA2.pem Adding debian:AffirmTrust_Premium.pem Adding debian:SZAFIR_ROOT_CA2.pem Adding debian:Amazon_Root_CA_4.pem Adding debian:Baltimore_CyberTrust_Root.pem Adding debian:Certigna.pem Adding debian:SSL.com_EV_Root_Certification_Authority_RSA_R2.pem Adding debian:Actalis_Authentication_Root_CA.pem Adding debian:Buypass_Class_3_Root_CA.pem Adding debian:emSign_ECC_RootCA-_C3.pem Adding debian:GLOBALTRUST_2020.pem Adding debian:QuoVadis_Root_CA_2_G3.pem Adding debian:AC_RAIZ_FNMT-RCM.pem Adding debian:USERTrust_RSA_Certification_Authority.pem Adding debian:AffirmTrust_Premium_ECC.pem Adding debian:SwissSign_SilverCA-_G2.pem Adding debian:AffirmTrust_Networking.pem Adding debian:DigiCert_Assured_ID_Root_G2.pem Adding debian:Entrust_Root_CertificationAuthority-_EC1.pem Adding debian:QuoVadis_Root_CA_3.pem done. Processing triggers for ca-certificates (20211016ubuntu0.20.04.1) ... Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d...

done.

boaks commented 1 year ago

Additional note:

When apt update and apt full-upgrade are executed, the system could be installed and works successfully.

blackboxsw commented 1 year ago

Thanks @boaks,

It looks like the package postinst failed for the install of ca-certificates-java due to some LD_PRELOAD env variable that doesn't seem to play well with whatever ca-certificates-java is doing in postinst. s I cannot reproduce this error launching Canonical's Ubuntu 20.04 on ec2 us-east-2 ami-06c4532923d4ba1ec. 

# no errors
ubuntu@ip-172-31-38-35:~$ cloud-init status --long --wait

status: done
boot_status_code: enabled-by-generator
last_update: Thu, 18 May 2023 17:11:14 +0000
detail:
DataSourceEc2Local

# jdk and ca-certs-java installed without error
ubuntu@ip-172-31-38-35:~$ dpkg -l | egrep 'openjdk-17-jre-headless|ca-certificates-java' 
ii  ca-certificates-java               20190405ubuntu1.1                 all          Common CA certificates (JKS keystore)
ii  openjdk-17-jre-headless:amd64      17.0.7+7~us1-0ubuntu1~20.04       amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
ubuntu@ip-172-31-38-35:~$ 

# snap installed
ubuntu@ip-172-31-38-35:~$ snap list
Name              Version        Rev    Tracking         Publisher     Notes
amazon-ssm-agent  3.1.1732.0     6312   latest/stable/…  aws✓          classic
certbot           2.6.0          3024   latest/stable    certbot-eff✓  classic
core18            20230426       2745   latest/stable    canonical✓    base
core20            20230503       1891   latest/stable    canonical✓    base
lxd               4.0.9-a29c6f1  24061  4.0/stable/…     canonical✓    -
snapd             2.59.2         19122  latest/stable    canonical✓    snapd
boaks commented 1 year ago

Do you run the cloud-config from my initial post? If not, would you provide me the cloud-config you're using?

blackboxsw commented 1 year ago

I launched with this cloud-config (I redacted the users: section you had as I deemed that irrelevant to the bug)

Here's the user-data I provided at launch

ubuntu@ip-172-31-38-35:~$ sudo cloud-init query userdata
#cloud-config
package_upgrade: true
packages:
# java - runtime for java application
 - openjdk-17-jre-headless
# fail2ban - network protection 
 - fail2ban
snap:
  commands:
   - snap refresh
# public x509 certificate / letsencrypt 
   - snap install --classic certbot
disable_root: false
boaks commented 1 year ago

Thanks! I spend also some time in testing:

Using AWS eu-central-1 with a newer ami (ami-0d497a49e7d359666) works. DigitalOcean "magically" works today (with out changes on my side) ExoScale works with a explicit template "Linux Ubuntu 20.04 LTS 64-bit"

Quite interesting, that even if package_upgrade: true is used, it seems to depend on the "age" of the image.

Anyway, thanks a lot for your help and pointing to the right direction (recent image).

igalic commented 1 year ago

I wonder if this was another eatmydata issue

boaks commented 1 year ago

eatmydata

I'm not common with that. I used ami-0d527b8c289b4af7f for eu-central-1. There was a new java-jdk release and that may have caused the trouble with the old images.

I'm now aware, that I need to take care of the really used images. My scripts have been improved. I'm happy with that result.