downstream issue with Amazon Root CAs

chris-brace commented 3 years ago

I am deploying an application that uses ex_aws which makes http requests using hackney which, in turn, depends on this repository. This application is happily deployed in seven app clusters and has been fine for two years now. Today, around 2.45-3.15pm EST, our ci/cd re-deployed the app due to a code change. The first thing the app does is make a call to https://sts.us-east-2.amazonaws.com/ which started failing with many of the following message:

{"time":"2021-03-16T21:03:39.038Z","severity":"warn","message":"ExAws: HTTP ERROR: {:tls_alert, {:unknown_ca, 'TLS client: In state certify at ssl_handshake.erl:1887 generated CLIENT ALERT: Fatal - Unknown CA\\n'}} for URL: \"https://sts.us-east-2.amazonaws.com/\" ATTEMPT: 9","metadata":{}}

The solution I found was to add ca-certificates package to the alpine image ( RUN apk update && apk add ca-certificates ) and tell hackney to use system certs!

config :ex_aws,
       :hackney_opts,
       ssl_options: [
         {:cacertfile,
          "/usr/share/ca-certificates/mozilla/Amazon_Root_CA_1.crt"}
       ],
       ssl_options: [
         {:cacertfile,
          "/usr/share/ca-certificates/mozilla/Amazon_Root_CA_2.crt"}
       ],
       ssl_options: [
         {:cacertfile,
          "/usr/share/ca-certificates/mozilla/Amazon_Root_CA_3.crt"}
       ],
       ssl_options: [
         {:cacertfile,
          "/usr/share/ca-certificates/mozilla/Amazon_Root_CA_4.crt"}
       ]

This is not ideal for obvious reasons and im worried something is badly broken.

benoitc commented 3 years ago

that should be fixed via 65f9eba5b1fb80caebc301d592b567a15d051f47 . I missed an update of the certificates.

chris-brace commented 3 years ago

thank you for all your work on these projects.

rlopzc commented 3 years ago

I fixed it by using: mix deps.update certifi

benoitc commented 3 years ago

it supposed to be fixed with latest version yes. I forgot to close the ticket ...

sashako commented 3 years ago

Hello, I'm not sure if it's the same issue or not - happy to open in a separate one. We were also experiencing unknown CA issues when we upgraded a bunch of dependencies. While some requests worked, some calls to aws iot were failing. We narrowed it down to certifi and had to downgrade to 2.5.2 to fix. Let me know if there's some other information I can provide on this.

benoitc commented 3 years ago

so using an old version of the bundle fixed it? this is odd. Which version were you using? On Thu 6 May 2021 at 16:21, Sasha Konovalova @.***> wrote:

Hello, I'm not sure if it's the same issue or not - happy to open in a separate one. We were also experiencing unknown CA issues when we upgraded a bunch of dependencies. While some requests worked, some calls to aws iot were failing. We narrowed it down to certifi and had to downgrade to 2.5.2 to fix. Let me know if there's some other information I can provide on this.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/certifi/erlang-certifi/issues/46#issuecomment-833563069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAADRIQYDZLNKGYYJNVOOK3TMKQXNANCNFSM4ZJPBQIA .

-- Sent from my Mobile

sashako commented 3 years ago

we were seeing issues on 2.6.1 and 2.6.0, 2.5.3 is what worked for us. But this is actually not an issue anymore! We resolved it by using AWS ATS endpoints instead of what we originally had and that worked like a charm with the new certifi version! Sorry for the confusion!

benoitc commented 2 years ago

closing the issue as fixed. use latest 2.8.0 version.

certifi / erlang-certifi

downstream issue with Amazon Root CAs #46