**Blocker**Certificate expiry with latest atsigncompany/virtualenv:dev_env image

[sitaram-kalluri]

Describe the bug Update the certs in at_server/at_functional_test/lib/secondary/base/certs with the latest certs and still seeing the certs expiry issue. This causes the failure of functional tests in at_client_sdk.

PR with which the certs are updated.

To Reproduce Steps to reproduce the behavior:

1. Pull the latest docker atsigncompany/virtualenv:dev_env image and run the container.
2. Run a plookup or lookup verb.
3. The server logs show the certificate expiry issue. Snippet from server logs: SEVERE|2022-02-15 14:27:07.904971|AtSecondaryServer|HandshakeException: Handshake error in server (OS Error: CERTIFICATE_VERIFY_FAILED: certificate has expired(handshake.cc:393))

Expected behavior Certificate expiry should not occur.

Additional context There is a change in verified By field between the old and new certs(Not sure if this is the cause; just an observation).

[gkc]

All certs are as they should be for this to work

Most likely a problem with verifying CA not being in the chain that the docker container sees

@cconstab any ideas on how to verify that (& fix it if so)?

[gkc]

The certs in the virtual environment image are the right ones, the image was pushed by https://github.com/atsign-foundation/at_server/actions/runs/1845269611

[gkc]

What I don't understand is why the functional tests are failing in at_client_sdk when they ran successfully in at_server, as the GitHub action builds the virtualenv docker image, and the tests are run against that docker image

[gkc]

OK I've downloaded the LetsEncrypt R3 CA cert Before adding it to cacert.pem:

gkc2019:certs gary$ openssl verify -CAfile cacert.pem cert.pem 
cert.pem: CN = vip.ve.atsign.zone
error 20 at 0 depth lookup:unable to get local issuer certificate

gkc2019:certs gary$ openssl verify -CAfile cacert.pem fullchain.pem 
cert.pem: CN = vip.ve.atsign.zone
error 20 at 0 depth lookup:unable to get local issuer certificate

After adding it to cacert.pem:

gkc2019:certs gary$ openssl verify -CAfile cacert.pem cert.pem 
cert.pem: OK

gkc2019:certs gary$ openssl verify -CAfile cacert.pem fullchain.pem 
fullchain.pem: OK

Created pull request #519

[gkc]

That PR didn't resolve this problem (although it’s “a” problem and might be a clue)

could it be the problem, except on the client rather than server side maybe? ie client machines not having that cert in their list of CA certs? However ... as I mentioned in a comment earlier, I don’t understand why the at_server functional tests are running ok, but the functional tests in at_client_sdk are not

[gkc]

@cconstab I'm out of ideas & out of steam for today. Maybe you can figure out what's going on here?

@cpswan I'm out Wed morning doing family stuff; if @cconstab hasn't had any luck, can you take a look?

[gkc]

The problem may be that the GitHub actions runner for at_client_sdk doesn’t have the latest list of root ca certs

• no cert errors for the at_server workflows which are working fine

• works fine on Sitaram's dev machine running 20.0.4

[gkc]

Estimated 3SP given combined time spent by Sita, Gary, Colin and Chris