department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
283 stars 204 forks source link

Jenkins OS check is broken #61523

Closed flooose closed 1 year ago

flooose commented 1 year ago

Issue Description

Jenkins os-check utility has been failing since 5/10/2023.

The output indicates that it is a connectivity issue with some of the instances. Inspecting these instances in EC2 seems to indicate, that an associated key-pair may be to blame.

Example output:

07:00:34  ok: [ip-172-31-1-68.us-gov-west-1.compute.internal]
07:00:35  fatal: [ip-172-31-8-205.us-gov-west-1.compute.internal]: UNREACHABLE! => changed=false 
07:00:35    msg: |-
07:00:35      Data could not be sent to remote host "ip-172-31-8-205.us-gov-west-1.compute.internal". Make sure this host can be reached over ssh: Warning: Permanently added 'ip-172-31-8-205.us-gov-west-1.compute.internal,172.31.8.205' (ECDSA) to the list of known hosts.
07:00:35      Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
07:00:35    unreachable: true
07:00:35  fatal: [ip-172-31-8-152.us-gov-west-1.compute.internal]: UNREACHABLE! => changed=false 
07:00:35    msg: |-
07:00:35      Data could not be sent to remote host "ip-172-31-8-152.us-gov-west-1.compute.internal". Make sure this host can be reached over ssh: Warning: Permanently added 'ip-172-31-8-152.us-gov-west-1.compute.internal,172.31.8.152' (ECDSA) to the list of known hosts.
07:00:35      Permission denied (publickey).
07:00:35    unreachable: true
07:00:35  [WARNING]: Unhandled error in Python interpreter discovery for host
07:00:35  ip-172-31-8-106.us-gov-west-1.compute.internal: Failed to connect to the host
07:00:35  via ssh: Warning: Permanently added 'ip-172-31-8-106.us-gov-
07:00:35  west-1.compute.internal,172.31.8.106' (ECDSA) to the list of known hosts.
07:00:35  Permission denied (publickey).
07:00:35  
07:00:36  [WARNING]: Unhandled error in Python interpreter discovery for host
07:00:36  ip-172-31-8-199.us-gov-west-1.compute.internal: Failed to connect to the host
07:00:36  via ssh: Warning: Permanently added 'ip-172-31-8-199.us-gov-
07:00:36  west-1.compute.internal,172.31.8.199' (ECDSA) to the list of known hosts.
07:00:36  Permission denied (publickey).
07:00:36  
07:00:36  fatal: [ip-172-31-8-106.us-gov-west-1.compute.internal]: UNREACHABLE! => changed=false 
07:00:36    msg: |-
07:00:36      Data could not be sent to remote host "ip-172-31-8-106.us-gov-west-1.compute.internal". Make sure this host can be reached over ssh: Warning: Permanently added 'ip-172-31-8-106.us-gov-west-1.compute.internal,172.31.8.106' (ECDSA) to the list of known hosts.
07:00:36      Permission denied (publickey).
07:00:36    unreachable: true
07:00:36  ok: [ip-172-31-8-17.us-gov-west-1.compute.internal]
07:00:36  ok: [ip-172-31-1-154.us-gov-west-1.compute.internal]
07:00:37  fatal: [ip-172-31-8-199.us-gov-west-1.compute.internal]: UNREACHABLE! => changed=false 
07:00:37    msg: |-
07:00:37      Data could not be sent to remote host "ip-172-31-8-199.us-gov-west-1.compute.internal". Make sure this host can be reached over ssh: Warning: Permanently added 'ip-172-31-8-199.us-gov-west-1.compute.internal,172.31.8.199' (ECDSA) to the list of known hosts.
07:00:37      Permission denied (publickey).
07:00:37    unreachable: true
07:00:38  ok: [ip-172-31-1-33.us-gov-west-1.compute.internal]
07:00:38  [WARNING]: Unhandled error in Python interpreter discovery for host
07:00:38  ip-172-31-10-172.us-gov-west-1.compute.internal: Failed to connect to the host
07:00:38  via ssh: Warning: Permanently added 'ip-172-31-10-172.us-gov-
07:00:38  west-1.compute.internal,172.31.10.172' (ECDSA) to the list of known hosts.
07:00:38  Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
07:00:38  
07:00:38  ok: [ip-172-31-11-130.us-gov-west-1.compute.internal]
07:00:38  [WARNING]: Unhandled error in Python interpreter discovery for host
07:00:38  ip-172-31-10-139.us-gov-west-1.compute.internal: Failed to connect to the host
07:00:38  via ssh: Warning: Permanently added 'ip-172-31-10-139.us-gov-
07:00:38  west-1.compute.internal,172.31.10.139' (ECDSA) to the list of known hosts.
07:00:38  Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
07:00:38  
07:00:39  fatal: [ip-172-31-10-172.us-gov-west-1.compute.internal]: UNREACHABLE! => changed=false 
07:00:39    msg: |-
07:00:39      Data could not be sent to remote host "ip-172-31-10-172.us-gov-west-1.compute.internal". Make sure this host can be reached over ssh: Warning: Permanently added 'ip-172-31-10-172.us-gov-west-1.compute.internal,172.31.10.172' (ECDSA) to the list of known hosts.
07:00:39      Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
07:00:39    unreachable: true
07:00:39  ok: [ip-172-31-3-191.us-gov-west-1.compute.internal]
07:00:39  ok: [ip-172-31-2-143.us-gov-west-1.compute.internal]
07:00:40  fatal: [ip-172-31-10-139.us-gov-west-1.compute.internal]: UNREACHABLE! => changed=false 
07:00:40    msg: |-
07:00:40      Data could not be sent to remote host "ip-172-31-10-139.us-gov-west-1.compute.internal". Make sure this host can be reached over ssh: Warning: Permanently added 'ip-172-31-10-139.us-gov-west-1.compute.internal,172.31.10.139' (ECDSA) to the list of known hosts.
07:00:40      Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
07:00:40    unreachable: true
07:00:40  ok: [ip-172-31-2-191.us-gov-west-1.compute.internal]
07:00:40  ok: [ip-172-31-2-8.us-gov-west-1.compute.internal]

Acceptance Criteria

alyssagallion commented 1 year ago

Verified good to close.