shipyard get actions failing SSL: CERTIFICATE_VERIFY_FAILED

[nagajagan]

Describe the bug shipyard get actions command is failing due to it can't make a secure connection with shipyard URL. The ingress certificates are working for iam.

Steps To Reproduce Maintain treasurmap version @ https://github.com/airshipit/treasuremap/commit/2227df4a8d60581974f49501265c0b8230fbf414 and follow the steps to bring up genesis node.

Expected behavior shipyard get actions/commit/configdocs commands should work on secure connection.

Environment

• Treasuremap version: 2227df4
• Treasuremp site type: cruiser
• Airshipctl version: 18.04.6 LTS (Bionic Beaver)
• Operating system: 4.15.0-167-generic
• Kernel version: v1.17.2
• Kubernetes version:
• Go version:
• Hypervisor level 0 (if applicable):
• Hardware specs (e.g. 4 vCPUs, 16GB RAM, bare metal vs VM):

Detailed steps to isolate the issue with shipyard

# docker run --rm --net=host -it --entrypoint /bin/bash -u 0:0 -w /target -v /root/airship:/target quay.io/airshipit/shipyard:e64f17b91bffb9b521f4dc6a4cbe693c3b7a9ad6-ubuntu_xenial
root@att5gc20:/target# curl -v https://shipyard-nc.att-5gcore.bete.ericy.com/
# server verification fails.

# docker run --rm --net=host -it --entrypoint /bin/bash -u 0:0 -w /target -v /root/airship:/target -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt quay.io/airshipit/shipyard:e64f17b91bffb9b521f4dc6a4cbe693c3b7a9ad6-ubuntu_xenial
root@att5gc20:/target# curl -v https://shipyard-nc.att-5gcore.bete.ericy.com/
# server verification succeeds because this ca certificates has 131 certificates.
#shipyard still failing.

Need your help to understand why curl is working good with these certificates and shipyard is not working with the same certificates.

# shipyard -v 5 get actions
HTTPSConnectionPool(host='shipyard-nc.att-5gcore.bete.ericy.com', port=443): Max retries exceeded with url: /api/v1.0/actions?verbosity=5 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
Error: Client Error: HTTPSConnectionPool(host='shipyard-nc.att-5gcore.bete.ericy.com', port=443): Max retries exceeded with url: /api/v1.0/actions?verbosity=5 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))

# curl -v https://shipyard-nc.att-5gcore.bete.ericy.com/
* Rebuilt URL to: https://shipyard-nc.att-5gcore.bete.ericy.com/
*   Trying 10.109.84.70...
* Connected to shipyard-nc.att-5gcore.bete.ericy.com (10.109.84.70) port 443 (#0)
* found 131 certificates in /etc/ssl/certs/ca-certificates.crt
* found 575 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: ingress-nc.att-5gcore.bete.ericy.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=SE,ST=Stockholm County,L=Stockholm,O=Ericsson AB,CN=ingress-nc.att-5gcore.bete.ericy.com
*        start date: Tue, 01 Feb 2022 00:00:00 GMT
*        expire date: Tue, 31 Jan 2023 23:59:59 GMT
*        issuer: C=US,O=DigiCert Inc,CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: shipyard-nc.att-5gcore.bete.ericy.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Wed, 30 Mar 2022 18:43:54 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< x-shipyard-req: ed1202e0-4475-4183-8681-37b858cf6d55
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: deny
<
* Connection #0 to host shipyard-nc.att-5gcore.bete.ericy.com left intact

[sirajyasin]

@nagajagan , can you please check this code and make sure your shipyard script is setting all the required parameters including (REQUESTS_CA_BUNDLE) https://github.com/airshipit/treasuremap/blob/2227df4a8d60581974f49501265c0b8230fbf414/tools/deployment/seaworthy-virt/airship_gate/lib/airship.sh#L16

If i understand correct, for the shipyard to make use of the cert, you have to mount it and provide the cert in the specific parameter (REQUESTS_CA_BUNDLE)

For your query about how the curl is working, the curl's certificate lookup path (CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt) found the cert as you have explicitly mounted there.

Please try to check and test with all required parameters.

[nagajagan]

We are still using https://github.com/airshipit/treasuremap/blob/2227df4a8d60581974f49501265c0b8230fbf414/tools/airship#L189

Is this deprecated? Could you tell me what is the difference between the two airship.sh files? Which one we are supposed to use? Is there any document to explain the flow?

[sirajyasin]

@nagajagan , can you please update if that helps and if we can mark this issue closed ?

[nagajagan]

We are working with cruiser type sites and changed treasuremap/tools/airship to include REQUESTS_CA_BUNDLE also and its working as a workaround for us now. Thank you @sirajyasin for the support.

[nagajagan]

site/xxxxx/secrets/certificates/ingress.yaml, ingress-crt-site to have following content and that should solve the problem.

-----BEGIN CERTIFICATE----- Ingress Certificates -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- Intermediate Certificate -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- Root certificate -----END CERTIFICATE-----

[nagajagan]

Issues is addressed by properly adding certificate chain in ingress.yaml