Athena failing on Babashka with InvalidSignatureException

[jmglov]

All requests to the Athena service seem to be failing with InvalidSignatureException. I'm using babashka v1.3.176 with a bb.edn like this:

{:deps {com.cognitect.aws/endpoints {:mvn/version "1.1.12.478"}
        com.cognitect.aws/athena {:mvn/version "847.2.1387.0"}
        com.grzm/awyeah-api {:git/url "https://github.com/grzm/awyeah-api"
                             :git/sha "9257dc0159640e46803d69210cae838d411f1789"
                             :git/tag "v0.8.41"}
        org.babashka/spec.alpha {:git/url "https://github.com/babashka/spec.alpha"
                                 :git/sha "8df0712896f596680da7a32ae44bb000b7e45e68"}}}

And am doing the following:

(require '[com.grzm.awyeah.client.api :as aws])
(import '(java.util UUID))

(def athena (aws/client {:api :athena, :region "eu-west-1"}))
(def request-token (str (UUID/randomUUID)))

(aws/invoke athena {:op :ListWorkGroups
                    :request {}})

This fails with:

{:__type "InvalidSignatureException"
 :message "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/\n\naccept:application/json\ncontent-type:application/x-amz-json-1.1\nhost:athena.eu-west-1.amazonaws.com:443\nx-amz-date:20230604T113541Z\nx-amz-target:AmazonAthena.ListWorkGroups\n\naccept;content-type;host;x-amz-date;x-amz-target\n44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20230604T113541Z\n20230604/eu-west-1/athena/aws4_request\na5ac7285ef28a4e83ce6bc8e1cc54820c55454e35dee67910b8d9607ca64d35e'\n"
 :cognitect.anomalies/category :cognitect.anomalies/incorrect}

I've tried various other Athena API calls, and they all fail with the same error. I've verified this works with the Cognitect client, using a deps.edn like this:

{:deps {com.cognitect.aws/api {:mvn/version "0.8.603"}
        com.cognitect.aws/endpoints {:mvn/version "1.1.12.478"}
        com.cognitect.aws/athena {:mvn/version "847.2.1387.0"}}}

I tried testing awyeah-api on JVM Clojure, but ran into #7 :sob:

[jmglov]

Verified that the same error occurs with JVM Clojure.

[jmglov]

I found the issue. Athena is expecting a canonical string like this:

POST
/

accept:application/json
content-type:application/x-amz-json-1.1
host:athena.eu-west-1.amazonaws.com:443
x-amz-date:20230605T111631Z
x-amz-target:AmazonAthena.ListWorkGroups

accept;content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a

and we're generating a canonical string like this:

POST
/

accept:application/json
content-type:application/x-amz-json-1.1
host:athena.eu-west-1.amazonaws.com
x-amz-date:20230605T111631Z
x-amz-target:AmazonAthena.ListWorkGroups

accept;content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a

The only difference is that our canonical string is missing the port in the host header. The expected canonical string has:

host:athena.eu-west-1.amazonaws.com:443

whilst we have:

host:athena.eu-west-1.amazonaws.com

I'll keep digging to see if I can understand why.

[stevebuik]

I've been getting this error message when using lambda get-function-configuration. Very confusing because the same operation works from the CLI.

digging deeper, it's only for lambda versions that have snapstart enabled i.e. the snapstart feature has only been a part of the aws api for 6 months or so.

I wonder if awyeah is out of sync with the latest aws api? could that cause Invalid signature exceptions across different apis?

I updated to latest commit: fb9bd3dbd0d87fe7dbb63424cbcca51b1fc4211b same error if using a :Version suffix in a lambda arn. could be a separate bug specific to lambda api but same error so might help here

I also updated all the aws deps to the latest versions but same error. That exhausts everything I can think of. Happy to help build a test case if it's useful since I have an operation that works from aws CLI but not using this lib. Just need some direction on how to proceed.

[jmglov]

I finally have some time to look into this more. @stevebuik thanks for your repro report. I'll try to get things working for Athena, and hopefully that will fix your issue as well. I'll ping you once I either have something or get completely stuck. :sweat_smile:

[grzm]

I've updated awyeah-api to catch up to aws-api https://github.com/cognitect-labs/aws-api/commit/5900e359365041ed9380947d010b0f0a853e793c

Let me know if the issue you're seeing persists.

[jmglov]

Still failing for me, tragically. I'll keep digging.

[stevebuik]

this update has fixed the exception I was getting using the Lambda api. thanks

@jmglov I enjoyed reading your blog post. seems like you will need to dig deep to solve this one

[grzm]

@stevebuik Glad to hear the updates fixed the issue.

[stevebuik]

the problem was similar to Josh'. when calling :GetFunctionConfiguration I needed to append a :Version onto the end of the fn name in the args. This :Version suffix is a newer feature and broke the request signature when used with the earlier lib.

Josh' blog post describes a similar thing where the Athena request has a slightly different shape.

[grzm]

I suspect the issue with host and port is related to how java.net.http.HttpClient handles host headers (https://github.com/grzm/awyeah-api/blob/main/docs/porting-decisions.markdown#javanethttphttpclient-java-11-java-12-java-17) and how that impacts signing.

[grzm]

On further investigation, appending the port number to the host header value in com.grzm.awyeah.signers/canonical-headers allows the athena invocation to succeed.

Open questions:

• I'd like to confirm that net.http.HttpClient is including the port number when it adds the host header to the request
• Determine what the correct fix is, which needs to ensure that it won't break other services. I suspect not, as I would think that the AWS service would be simple/dumb (by design).

I'd like the fix to ensure there's no implicit coupling between the com.grzm.awyeah.http-client implementation (which handles the host header itself) and adding the port to the host header value in signers; then again, I've already made this implicit assumption by not adding the host header elsewhere.

[grzm]

Well, it looks like by default the HttpClient sends only an authority header, not a host header.

Nov 20, 2023 8:09:54 PM jdk.internal.net.http.Http2Connection encodeHeaders
INFO: HEADERS: HEADERS FRAME (stream=1)
    :authority: athena.us-east-1.amazonaws.com:443
    :method: POST
    :path: /
    :scheme: https
    content-length: 2
    User-Agent: Java-http-client/21.0.1
    accept: application/json
    authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/us-east-1/athena/aws4_request, SignedHeaders=accept;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=[REDACTED]
    content-type: application/x-amz-json-1.1
    x-amz-date: 20231121T020953Z
    x-amz-security-token: [REDACTED]
    x-amz-target: AmazonAthena.ListWorkGroups1

Assuming the logs are telling the tale, the AWS service is likely taking the authority header as the host header.

For those playing along at home, I set jdk.httpclient.HttpClient.log and jdk.internal.httpclient.info System properties.

   :jvm-opts ["-Djdk.httpclient.HttpClient.log=errors,requests,headers"
              "-Djdk.internal.httpclient.info=true"]

[grzm]

@jmglov Try HEAD. (I haven't tagged or updated the README yet):

com.grzm/awyeah-api {:git/url "https://github.com/grzm/awyeah-api"
                     :git/sha "e5513349a2fd8a980a62bbe0d45a0d55bfcea141"}

Some AWS services (including athena) support HTTP/2, which uses the :authority pseudo header to represent the authority of the URI (which includes the host and port) instead of the HOST header used in HTTP/1.1. AWS is likely mapping the :authority header value to HOST when validating the request signature. By forcing the http-client to use HTTP/1.1, we avoid this behavior.

The alternative of selectively appending port to the host header of some services seems brittle: likely HTTP/2 support in AWS services is a moving target.

[jmglov]

@grzm Sorry for the long delay. :grimacing:

I tested it and it worked! :tada:

(ns athena
  (:require [com.grzm.awyeah.client.api :as aws]))

(comment

  (def athena (aws/client {:api :athena, :region "eu-west-1"}))
  ;; => #'athena/athena

  (aws/invoke athena {:op :ListWorkGroups
                      :request {}})
  ;; => {:WorkGroups
  ;;     [{:Name "primary",
  ;;       :State "ENABLED",
  ;;       :Description "",
  ;;       :CreationTime #inst "2023-06-04T11:35:15.000-00:00",
  ;;       :EngineVersion
  ;;       {:SelectedEngineVersion "AUTO",
  ;;        :EffectiveEngineVersion "Athena engine version 3"}}]}

  )

[grzm]

Thanks for confirming! Tagged and README updated. Thanks for the detailed report. The example was particularly helpful.

[jmglov]

Sure thing! Thanks for fixing it! :slightly_smiling_face: