Credential procurement doesn't work on Lambda

[Nexuist]

Hi,

I have a bit of a unique use case in that I am using the aws package on Lambda by compiling my code with raco exe and running the resulting binary using Node's child_process.

I have gotten everything working except that the signature v4 method fails to build valid headers and I get an unfortunate error:

{"__type":"com.amazon.coral.service#UnrecognizedClientException","message":"The security token included in the request is invalid."}

Now, here is my dilemma. Initially I was using use-iam-ec2-credentials! to grab the keys from the metadata service and use those for signing. Unfortunately the metadata service does not exist to the Lambda container. As a result that method ends with a tcp-connect failure.

So, onto the alternative. Lambda provides all three creds (public key, private key, session token) as environment variables, so I should just be able to pull them out, right?

And I kind of am, except I can't pull out the session token. And that's the problem. Take a look at the method in aws/keys.rkt. It only embeds the X-Amz-Security-Token (using the session token) if iam-role is set - and I can't set iam-role or pull the-creds because the metadata service doesn't exist, and that crashes everything!

Suggested fix: make session-token a parameter just like private-key and public-key are so I can set it manually from the environment variables. I think that would fix my problems.

Thanks for taking the time to read this, looking forward to your response :)

P.S. thank you for this library. I've had a blast building a DynamoDB ORM that I hope to open source sometime soon!

[Nexuist]

I've been able to build a simple replacement for now until I can use the regular methods again:

#lang racket/base

(require racket/contract)
(require racket/dict)
(require http/request)
(require json)

(require aws/util)
(require aws/exn)
(require aws/keys)
(require aws/sigv4)
(require aws/dynamo)

(define session-token (make-parameter ""))
(define environment-tokens? (string? (getenv "AWS_SESSION_TOKEN")))

(if environment-tokens?
  (begin
    (public-key (getenv "AWS_ACCESS_KEY_ID"))
    (private-key (getenv "AWS_SECRET_ACCESS_KEY"))
    (session-token (getenv "AWS_SESSION_TOKEN")))
  (read-keys/aws-cli))

(define/contract (date+authorize method uri heads body)
  (-> string? string? dict? bytes? any)
  (let ([heads (dict-set* heads
                          'Host (endpoint-host (dynamo-endpoint))
                          'Date (seconds->gmt-8601-string 'basic
                                                          (current-seconds))
                          'Content-Type "application/x-amz-json-1.0")])
    (if (not (string=? (session-token) ""))
      (set! heads (dict-set* heads 'X-Amz-Security-Token (session-token)))
      (void))
    (add-v4-auth-heads #:heads heads
                       #:method method
                       #:uri uri
                       #:sha256 (sha256-hex-string body)
                       #:region (dynamo-region)
                       #:service "dynamodb")))

(define (target op-name)
  (string-append "DynamoDB_" (dynamo-api-version) "." op-name))

(define/contract (dynamo-custom-request op js)
  (-> string? jsexpr? jsexpr?)
  (define p "/")
  (define u (endpoint->uri (dynamo-endpoint) p))
  (define bstr (string->bytes/utf-8 (jsexpr->string js)))
  (define h (date+authorize "POST"
                            u
                            (hash 'x-amz-target (target op))
                            bstr))
  (call/output-request "1.1" "POST" u bstr (bytes-length bstr) h
                       (λ (in h)
                         (check-response in h)
                         (bytes->jsexpr (read-entity/bytes in h)))))

(provide dynamo-custom-request)

I basically just ripped out some of the methods in aws/dynamo and rewrote them to use thesession-token parameter. It also reads the keys from environment variables now.

Making session-token a parameter and perhaps supplying a (read-keys/from-environment) function may be the best bet now.

[greghendershott]

Thanks for reporting this and sharing a work-around.

What you suggest seems right. But I'd want to make sure adding this third scenario (AWS Lambda) wouldn't break the other two ("normal", and, EC2 instance metadata). Unfortunately I don't have time to do that confidently, now. [Plus my Racket time is on "pause", now.]. So I will leave this open, for now.

[Nexuist]

@greghendershott Understood, I read your article previously and I respect your choice. Best of luck with future endeavors!

[greghendershott]

@Nexuist So I found a little time to look at this. I pushed a commit -- not yet merged to master. If you have a chance to look at it, that would be great.

The idea is that if you are running on Lambda, you call the new read-keys-and-token/environment function, much as you suggested.

Note that my commit will automatically handle all the various services like dynamo.rkt because they use add-v4-auth-heads which in turn calls ensure-ec2-instance-credentials-and-add-token-header which I have now extended to handle the case where the new security-token parameter is non-false as a result of read-keys-and-token/environment.

I also tried to update the docs to clarify the three uses cases ("plain", EC2 instance, and Lambda).

What I have not done is test any of this -- not on Lambda or to make sure the old things still work.

[greghendershott]

p.s. A dumb question because I have zero mileage using AWS Lamda: Do the environment variables ever change -- might the params get "stale" and we should re-read them periodically? Or is that N/A because the AWS Lambda containers get discarded and recreated frequently?

[greghendershott]

I went on to do quite a bit more work. Just squashed to commit b433002, which replaces the previous one I mentioned.

[Nexuist]

Do the environment variables ever change -- might the params get "stale" and we should re-read them periodically?

That's a great question. I do not handle this case in my solution. I believe you are correct that there is no need to periodically refresh the keys because the containers do get discarded and recreated frequently (I believe generally after 15 minutes the container dies, but it may stay alive longer if it is being bombarded by requests. Also likely is that Lambda spins up another container to replace an aging one).

I don't think there is any documentation stating that the environment variables get refreshed at any time. If you need to refresh credentials, I think you need to do it through STS, which is how the AWS Node.js SDK handles it. If any request fails due to stale tokens, it tries getting new tokens and resending the request.

In my opinion this is a lot of overhead and you are fine just using the tokens provided. If it does become an issue, I'll let you know :)

As for testing, I will soon begin the process of open sourcing my ORM which will probably involve building another Lambda. I can try using your branch for that. Is there a way to tell raco to switch branches or will I have to clone the repo manually?

[greghendershott]

I don't think there is any documentation stating that the environment variables get refreshed at any time. If you need to refresh credentials, I think you need to do it through STS, which is how the AWS Node.js SDK handles it. If any request fails due to stale tokens, it tries getting new tokens and resending the request.

In the case of EC2 instance metadata, not only are the keys and token provided, so is an explicit expiration value -- as well as a guarantee that re-reading will get new/refreshed credentials at least 5 minutes before expiration.

If Lambda were to supply credentials initially through env vars ... let them expire but provide no expiration time ... and furthermore, require you to get subsequent refreshed credentials using some whole other method than env vars, like STS? That would be very weird and make me very sad. I guess anything is possible; maybe let's just cross that bridge when we come to it.

As for testing, I will soon begin the process of open sourcing my ORM which will probably involve building another Lambda. I can try using your branch for that. Is there a way to tell raco to switch branches or will I have to clone the repo manually?

I actually went ahead and merged to master. That didn't seem incredibly reckless because it's brand-new functionality. Worst-case it might not work and will need a fix -- but it shouldn't break existing code.

[greghendershott]

p.s. The update is already on the Racket package server, so raco will now use it. (Unfortunately the docs page there hasn't refreshed yet; should happen w/in 24 hours. The TL;DR is to call the new (credentials-from-environment!) once as initialization.)

[Nexuist]

Here's a quick article I found: https://alestic.com/2014/11/aws-lambda-environment/

Here's the only docs I can find on provided env variables: https://docs.aws.amazon.com/lambda/latest/dg/lambda-environment-variables.html

It looks like there is no expiration value. Perhaps Lambda tokens are special in that they do not expire based on time? I'll have to explore.

Also, noted. I'll just grab it from Racket. Thanks!