IMDSv2 not supported

[burbma]

Amazon Linux 2023 does not allow use of IMDSv1 but rather requires use of IMDSv2 (see here for the difference). cognitect.aws.ec2-metadata-utils only allows use of v1. This means aws-api cannot authenticate with Instance Profile Credentials (number 6 in the credential provider chain, as of this writing, here).

Dependencies

{:deps {com.cognitect.aws/api {:mvn/version "0.8.686"}}}

Description with failing test case

First, be on an ec2 running Amazon Linux 2023. Then,

(require '[cognitect.aws.ec2-metadata-utils :refer :all])
(require '[cognitect.aws.http.cognitect :as http])
(def http-client (http/create))
(get-ec2-instance-data http-client)
;; See that this returns nil

There are two problems here. First, it didn't return instance metadata. Second, it returned nil rather than failing. If you look this line you'll see it's swallowing any errors from this request. When I run the http request from these lines manually to inspect the response I see this following swallowed error

{:status 401, :headers {"server" "EC2ws", "connection" "close", "content-length" "0", "date" "Sat, 07 Oct 2023 02:59:57 GMT", "content-type" "text/plain"}, :body nil, :cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message "HTTP protocol violation: Authentication challenge without WWW-Authenticate header", :cognitect.http-client/throwable #error {
 :cause "HTTP protocol violation: Authentication challenge without WWW-Authenticate header"
 :via
 [{:type org.eclipse.jetty.client.HttpResponseException
   :message "HTTP protocol violation: Authentication challenge without WWW-Authenticate header"
   :at [org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener onComplete "AuthenticationProtocolHandler.java" 164]}]
 :trace
 [[org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener onComplete "AuthenticationProtocolHandler.java" 164]
  [org.eclipse.jetty.client.ResponseNotifier notifyComplete "ResponseNotifier.java" 218]
  [org.eclipse.jetty.client.ResponseNotifier notifyComplete "ResponseNotifier.java" 210]
  [org.eclipse.jetty.client.HttpReceiver terminateResponse "HttpReceiver.java" 481]
  [org.eclipse.jetty.client.HttpReceiver terminateResponse "HttpReceiver.java" 461]
  [org.eclipse.jetty.client.HttpReceiver responseSuccess "HttpReceiver.java" 424]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP messageComplete "HttpReceiverOverHTTP.java" 374]
  [org.eclipse.jetty.http.HttpParser handleContentMessage "HttpParser.java" 597]
  [org.eclipse.jetty.http.HttpParser parseContent "HttpParser.java" 1668]
  [org.eclipse.jetty.http.HttpParser parseNext "HttpParser.java" 1551]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP parse "HttpReceiverOverHTTP.java" 208]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP process "HttpReceiverOverHTTP.java" 148]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP receive "HttpReceiverOverHTTP.java" 80]
  [org.eclipse.jetty.client.http.HttpChannelOverHTTP receive "HttpChannelOverHTTP.java" 131]
  [org.eclipse.jetty.client.http.HttpConnectionOverHTTP onFillable "HttpConnectionOverHTTP.java" 172]
  [org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded "AbstractConnection.java" 311]
  [org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 105]
  [org.eclipse.jetty.io.ChannelEndPoint$1 run "ChannelEndPoint.java" 104]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill runTask "EatWhatYouKill.java" 338]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill doProduce "EatWhatYouKill.java" 315]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill tryProduce "EatWhatYouKill.java" 173]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill produce "EatWhatYouKill.java" 137]
  [org.eclipse.jetty.util.thread.QueuedThreadPool runJob "QueuedThreadPool.java" 883]
  [org.eclipse.jetty.util.thread.QueuedThreadPool$Runner run "QueuedThreadPool.java" 1034]
  [java.lang.Thread run "Thread.java" 833]]}}

Indeed IMDSv2 requires a form of authentication that v1 does not.

[scottbale]

Thanks for the very thorough writeup!

[tjg]

Hi! For anyone curious, here's how I got it working under IMDSv2

Maybe someting like it could be mentioned in the README, so people don't have to stub their toes on this problem?

I'd be grateful for any ruthless criticism. Like:

• Should I use retries?
• How to handle failures in CredentialsProvider's .fetch method? (Return nil, throw an exception, etc?)

Soooo, I can declare:

(def s3
  (aws/client {:api :s3
               :region "us-east-1"
               :credentials-provider (role-credentials-provider)}))

Using:

(ns test-test-test.aws-api-auth
  (:require
   [clj-http.client :as client]
   [clojure.data.json :as json]
   [cognitect.aws.client.api :as aws]
   [cognitect.aws.credentials :as credentials]))

(def ^:private token-url
  "http://169.254.169.254/latest/api/token")

(def ^:private security-credentials-url
  "http://169.254.169.254/latest/meta-data/iam/security-credentials/")

(defn- get-imdsv2-token []
  (let [{:keys [body status] :as response}
        (client/put token-url
                    {:headers {"X-aws-ec2-metadata-token-ttl-seconds" 
                               "21600"}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IMDSv2 token available." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-name [token]
  (let [{:keys [body status] :as response}
        (client/get security-credentials-url
                    {:headers {"X-aws-ec2-metadata-token" token}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IAM role found." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-credentials
  ([token]
   (let [role-name (get-iam-role-name token)]
     (when role-name
       (get-iam-role-credentials token role-name))))
  ([token role-name]
   (let [{:keys [body status] :as response}
         (client/get (str security-credentials-url role-name)
                     {:headers {"X-aws-ec2-metadata-token" token}
                      :throw-exceptions false})]
     (if (= 200 status)
       (json/read-str body :key-fn keyword)
       (throw (ex-info "No IAM role credentials found." 
                       {:IMDSv2-status-code status}))))))

(defn- get-credentials []
  (some->> (get-imdsv2-token)
           (get-iam-role-credentials)))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Public API

(defn role-credentials-provider []
  (credentials/cached-credentials-with-auto-refresh
   (reify credentials/CredentialsProvider
     (fetch [_]
       (when-let [creds (try
                          (get-credentials)
                          ;; Err... what happens if I throw in here?
                          ;; Or return nil?
                          (catch Exception e))]
         {:aws/access-key-id     (:AccessKeyId creds)
          :aws/secret-access-key (:SecretAccessKey creds)
          :aws/session-token     (:Token creds)
          ::credentials/ttl      (credentials/calculate-ttl creds)})))))

[brandonstubbs]

Hi

It seems there is another ticket requesting IMDSv2 support #165

While Amazon Linux 2023 uses IMDSv2 by default (Which you should use as best practice) you can still change this. https://docs.aws.amazon.com/linux/al2023/ug/compare-with-al2.html#imdsv2

For example, launching a new instance via cli

aws ec2 run-instances \
    ...
    --metadata-options "HttpTokens=optional"

Or via the console in Launch an instance (Advanced details):

[tlonist-sang]

Hi! For anyone curious, here's how I got it working under IMDSv2

Maybe someting like it could be mentioned in the README, so people don't have to stub their toes on this problem?

I'd be grateful for any ruthless criticism. Like:

• Should I use retries?
• How to handle failures in CredentialsProvider's .fetch method? (Return nil, throw an exception, etc?)

Soooo, I can declare:

(def s3
  (aws/client {:api :s3
               :region "us-east-1"
               :credentials-provider (role-credentials-provider)}))

Using:

(ns test-test-test.aws-api-auth
  (:require
   [clj-http.client :as client]
   [clojure.data.json :as json]
   [cognitect.aws.client.api :as aws]
   [cognitect.aws.credentials :as credentials]))

(def ^:private token-url
  "http://169.254.169.254/latest/api/token")

(def ^:private security-credentials-url
  "http://169.254.169.254/latest/meta-data/iam/security-credentials/")

(defn- get-imdsv2-token []
  (let [{:keys [body status] :as response}
        (client/put token-url
                    {:headers {"X-aws-ec2-metadata-token-ttl-seconds" 
                               "21600"}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IMDSv2 token available." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-name [token]
  (let [{:keys [body status] :as response}
        (client/get security-credentials-url
                    {:headers {"X-aws-ec2-metadata-token" token}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IAM role found." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-credentials
  ([token]
   (let [role-name (get-iam-role-name token)]
     (when role-name
       (get-iam-role-credentials token role-name))))
  ([token role-name]
   (let [{:keys [body status] :as response}
         (client/get (str security-credentials-url role-name)
                     {:headers {"X-aws-ec2-metadata-token" token}
                      :throw-exceptions false})]
     (if (= 200 status)
       (json/read-str body :key-fn keyword)
       (throw (ex-info "No IAM role credentials found." 
                       {:IMDSv2-status-code status}))))))

(defn- get-credentials []
  (some->> (get-imdsv2-token)
           (get-iam-role-credentials)))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Public API

(defn role-credentials-provider []
  (credentials/cached-credentials-with-auto-refresh
   (reify credentials/CredentialsProvider
     (fetch [_]
       (when-let [creds (try
                          (get-credentials)
                          ;; Err... what happens if I throw in here?
                          ;; Or return nil?
                          (catch Exception e))]
         {:aws/access-key-id     (:AccessKeyId creds)
          :aws/secret-access-key (:SecretAccessKey creds)
          :aws/session-token     (:Token creds)
          ::credentials/ttl      (credentials/calculate-ttl creds)})))))

Thanks for sharing your code! I think returning nil makes more sense here, especially when you want to include this provider in chain-credentials-provider. Inside of it is a function called valid-credentials that checks if the fetched credentials is valid. The creds having nil will be handled by the function.

[dharrigan]

This bit me hard today on a production system (causing production issues). Setting it to be "optional" in the console got round this problem (for now! - we have an automated deployment system, so have to now figure out how to shove this into the metadata setup of the ec2 instance).

Please can support for IMDSv2 within the codebase be considered as soon as possible.

Thank you.

-=david=-

[zerg000000]

I think the region provider has a similar issue, but it should have its own ticket.

[scottbale]

We have just released a beta release which should solve this issue.

@burbma if time permits, could you please verify whether this release solves your issue? Thanks.

(I will leave this issue open for the time being until we verify.)

[eduardozanotto]

@scottbale it probably fixed now

I'm using the awyeah lib in my project and found the same problem of not getting the correct credentials from AWS. I found this issue and tried "purely" using only cognitect.aws.api, and I didn't have the same problem. It returned my credentials and the assumed role.

user=> (aws-ya/invoke sts-ya {:op :GetCallerIdentity})
#:cognitect.anomalies{:category :cognitect.anomalies/fault, :message "Unable to fetch credentials. See log for more details."}
user=> (aws-cog/invoke sts-cog {:op :GetCallerIdentity})
{:UserId "*****", :Account "*****", :Arn "*******"}
user=>

[burbma]

@scottbale Sorry I just saw this. Unfortunately I no longer work for the company where I was encountering this issue and so I can't reproduce my issue as it existed and that context is long gone from my brain. Any tests done by others will be just as effective as anything I could do.