Error Building CMS Application in Jenkins

[7hunderbird]

User Story or Problem Statement

The build job that prepares the base machine image for the CMS deploy is failing to complete.

Description or Additional Context

It appears that this is due to the datadog gpg key expiring.

• https://github.com/department-of-veterans-affairs/devops/pull/14540

[7hunderbird]

• commits on main branch
  • fe3c0883f2cf450f45bf61cf5ba83ccb85f45d8b
  • eacfeb85de096d23071fa462f60b99b39290e487
  • e34d10997222718348a649a212451fbf6c6cb580
  • fc2f79d312616ceca2398c06c2d6ceef95f05948
  • 2b4b7e10ccba02df0a99e8a924e5002f611462b7 v0.0.1008
  • 4b78a606f586f8627dc68fa1e1a176097950374b
  • de3a8d9274261cd2e595087f70b720c07a505580
  • 4c36d789edc91986742093792eea93110ff35957
  • 43a1965b85a44d1d97af5ea7e27618c3037a5a84
  • 0e683c033a8a17e62e46a545ecc6dba808620bce
  • a486d4c26c7a8b293b5de05d855a1d763211426a
  • 0e3d7e40f040dc9dd127ab52e9696d2fd50ab503 v0.0.1007
• staging.cms.va.gov currently deployed to 2b4b7e10c
  • https://staging.cms.va.gov/user/4948
• prod.cms.va.gov currently deployed to v0.0.1007
  • https://prod.cms.va.gov/user/4948

[7hunderbird]

The test job calls a build. It's set to retry 3 times.

At this time it's failing 3 different ways:

1. Wait for cloud-init to complete
2. Install latest datadog-agent package
3. Check out Drupal to the docroot

Wait for cloud-init to complete

• It appears that it attempts to make an SSH connection to the new AMI (Amazon Machine Image) and fails to connect because "Permission denied".
• In the next two builds, the Wait for cloud-init to complete connects without issue
• Since it works 66% of the time, we'll not choose that as the highest priority thread to troubleshoot.

Check out Drupal to the docroot

• While I intially thought this issue would be the best to focus on, Tim pointed out today that this task does run successfully in the one where Install latest datadog-agent package fails

Install latest datadog-agent package

• Initial troubleshooting will focus on this

---

Individual Failures

• http://jenkins.vfs.va.gov/job/builds/job/cms/4903/consoleFull

16:22:13  TASK [_launch-bake : Wait for cloud-init to complete] **************************
16:22:13  Monday 15 July 2024  16:22:12 +0000 (0:00:10.308)       0:00:25.260 *********** 
16:22:13  [WARNING]: Unhandled error in Python interpreter discovery for host localhost:
16:22:13  Failed to connect to the host via ssh: Warning: Permanently added
16:22:13  'ec2-3-31-59-28.us-gov-west-1.compute.amazonaws.com,3.31.59.28' (ECDSA) to the
16:22:13  list of known hosts.  Permission denied (publickey,gssapi-keyex,gssapi-with-
16:22:13  mic).
16:22:13  
16:22:14  fatal: [localhost]: UNREACHABLE! => changed=false 
16:22:14    msg: |-
16:22:14      Data could not be sent to remote host "ec2-3-31-59-28.us-gov-west-1.compute.amazonaws.com". Make sure this host can be reached over ssh: Warning: Permanently added 'ec2-3-31-59-28.us-gov-west-1.compute.amazonaws.com,3.31.59.28' (ECDSA) to the list of known hosts.
16:22:14      Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
16:22:14    unreachable: true
16:22:14

• http://jenkins.vfs.va.gov/job/builds/job/cms/4904/consoleFull

  16:41:20  TASK [datadog : Install latest datadog-agent package] **************************
  16:41:20  Monday 15 July 2024  16:41:20 +0000 (0:00:00.056)       0:18:25.095 *********** 
  16:41:42  fatal: [ec2-3-31-59-28.us-gov-west-1.compute.amazonaws.com]: FAILED! => changed=true 
  16:41:42    changes:
  16:41:42      installed:
  16:41:42      - datadog-agent
  16:41:42      updated: []
  16:41:42    msg: |-
  16:41:42      warning: /var/cache/yum/x86_64/2/ansible_datadog_7/packages/datadog-agent-7.55.1-1.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b01082d3: NOKEY
  16:41:42    
  16:41:42    
  16:41:42      The GPG keys listed for the "Datadog, Inc." repository are already installed but they are not correct for this package.
  16:41:42      Check that the correct key URLs are configured for this repository.
  16:41:42    
  16:41:42    
  16:41:42       Failing package is: 1:datadog-agent-7.55.1-1.x86_64
  16:41:42       GPG Keys are configured as: [https://s3.amazonaws.com/public-signing-keys/DATADOG_RPM_KEY_FD4BF915.public](https://s3.amazonaws.com/public-signing-keys/DATADOG_RPM_KEY_FD4BF915.public%1B[0m)
  16:41:42    obsoletes:
  16:41:42      python-lockfile:
  16:41:42        dist: noarch
  16:41:42        repo: installed
  16:41:42        version: 1:0.9.1-4.amzn2
  16:41:42      python-simplejson:
  16:41:42        dist: x86_64
  16:41:42        repo: installed
  16:41:42        version: 3.2.0-1.amzn2.0.2
  16:41:42    rc: 1
  16:41:42    results:
  16:41:42    - |-
  16:41:42      Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
  16:41:42      235 packages excluded due to repository priority protections
  16:41:42      Resolving Dependencies
  16:41:42      --> Running transaction check
  16:41:42      ---> Package datadog-agent.x86_64 1:7.55.1-1 will be installed
  16:41:42      --> Finished Dependency Resolution
  16:41:42    
  16:41:42      Dependencies Resolved
  16:41:42    
  16:41:42      ================================================================================
  16:41:42       Package            Arch        Version            Repository              Size
  16:41:42      ================================================================================
  16:41:42      Installing:
  16:41:42       datadog-agent      x86_64      1:7.55.1-1         ansible_datadog_7      259 M
  16:41:42    
  16:41:42      Transaction Summary
  16:41:42      ================================================================================
  16:41:42      Install  1 Package
  16:41:42    
  16:41:42      Total download size: 259 M
  16:41:42      Installed size: 259 M
  16:41:42      Downloading packages:
  16:41:42      Public key for datadog-agent-7.55.1-1.x86_64.rpm is not installed
  16:41:42      Retrieving key from [https://s3.amazonaws.com/public-signing-keys/DATADOG_RPM_KEY_FD4BF915.public](https://s3.amazonaws.com/public-signing-keys/DATADOG_RPM_KEY_FD4BF915.public%1B[0m)
  16:41:42

• http://jenkins.vfs.va.gov/job/builds/job/cms/4905/consoleFull

  16:47:03  TASK [build/roles/cms/galaxy/geerlingguy.drupal : Check out Drupal to the docroot.] ***
  16:47:03  Monday 15 July 2024  16:47:03 +0000 (0:00:00.030)       0:04:35.212 *********** 
  16:47:10  fatal: [ec2-3-31-59-28.us-gov-west-1.compute.amazonaws.com]: FAILED! => changed=false 
  16:47:10    cmd: /bin/git reset --hard HEAD
  16:47:10    msg: |-
  16:47:10      error: unable to unlink old 'docroot/sites/default/settings.php': Permission denied
  16:47:10      fatal: Could not reset index file to revision 'HEAD'.
  16:47:10    rc: 128
  16:47:10    stderr: |-
  16:47:10      error: unable to unlink old 'docroot/sites/default/settings.php': Permission denied
  16:47:10      fatal: Could not reset index file to revision 'HEAD'.
  16:47:10    stderr_lines: <omitted>
  16:47:10    stdout: ''
  16:47:10    stdout_lines: <omitted>
  16:47:10  

[7hunderbird]

I think I "closed" this when I meant to select "Comment" to save changes.