Documentation improvement suggestions

[jessemyers]

I am trying to run aws-sigv4-proxy in front of AWS-hosted prometheus (aps). After a lot of experimentation, I found something that works; it would be great if the documentation were clearer about the places where I went wrong.

• The signature mechanism appears not to support session tokens. I took the documentation that suggested mounting ~/.aws into the container and setting AWS_PROFILE to mean that the proxy was using the standard AWS tool chain for interacting with credentials. Since our setup leans heavily on role assumption and SSO, the path of least resistance (and best security) involves setting AWS_SESSION_TOKEN and this does not work. I only made progress once I created an ad hoc AWS user in the target account and created a temporary access token.
• The proxy does not do anything smart with HTTP headers provided by the user agent. In my case, I was using a CLI http client that set Accept, Accept-Heading, Connection, Host, and User-Agent. My signature failed until I explicitly skipped all of these headers. (It's possible that I only needed to skip a subset... I don't know the signature protocol very well, which is why I was trying to use the proxy in the first place)

In any case, my working configuration ended up using:

docker run --rm -it \
  -p 8080:8080 \
  --env-file .env \
  public.ecr.aws/aws-observability/aws-sigv4-proxy \
  --region us-west-2 \
  --name aps \
  --host aps-workspaces.us-west-2.amazonaws.com \
  --strip Accept \
  --strip Accept-Encoding \
  --strip Connection \
  --strip Host \
  --strip User-Agent \
  --verbose --log-failed-requests --log-signing-process

where .env contained:

AWS_ACCESS_KEY_ID=<REDACTED>
AWS_SECRET_ACCESS_KEY=<REDACTED>

[alvinlin123]

Hi @jessemyers

I happen to work in the AWS-hosted Prometheus team. I am curious about your use case on why you still need this proxy because we have added AWS SigV4 support to Grafana as well as the open source Prometheus.

With that said, let's go back to your original question. The AWS_SESSION_TOKEN should work, I will look into why this didn't work.

You are right about proxy does not do anything smart about HTTP headers, it pretty much sign all the headers you give. The case where signature would fail is if the header value changes between when you sign it and when AWS server receives it. I am wondering if your Connection header value gets modified by the proxies between your machine and AWS server. To illustrate this issue, let me give an example. The X-Forwarded-For header should not be part of the Sigv4 signature because the headers value get modified by load balancers hop to hop.

[jessemyers]

I need this proxy because sig4v appears to be the only way to connect to AWS Prometheus. As you might imagine there are valid reasons to want to query Prometheus from something that is NOT Grafana, especially when debugging. (What good is an API if you can't call it from a client of your choice?)

[alvinlin123]

I was able to reproduce the AWS_SESSION_TOKEN issue, even if I give the AWS_SDK_LOAD_CONFIG env variable, the aws cli profile loading was failing.

I switched the base image from scratch to alpine: latest, and if I set the AWS_SDK_LOAD_CONFIG true then the aws cli profiles was loaded successfully.

@jessemyers would you be able to kindly do this quick test for me? Will you be able to build the changes in this PR https://github.com/awslabs/aws-sigv4-proxy/pull/57, specify env var AWS_SDK_LOAD_CONFIG=true when running the image, and see if you can load the credentials under your ~/.aws?

[jessemyers]

I might be doing something wrong, but I still see errors.

Here's what I did...

1. Build the image

git clone git@github.com:alvinlin123/aws-sigv4-proxy.git
cd aws-sigv4-proxy
git co sratch-doesnt-seem-to-te-able-to-load-aws-cli-profile
docker build -t aws-sig4v-proxy .

2. AWS sso Login

aws --profile ${PROFILE_NAME} sso login

3. Run the container

docker run -it --rm \
    -p 8080:8080 \
    --env AWS_SDK_LOAD_CONFIG=true \
    --env AWS_PROFILE=${PROFILE_NAME}  \
    -v ~/.aws:/root/.aws \
    aws-sigv4-proxy \
    --region us-west-2 \
    --name aps \
    --host aps-workspaces.us-west-2.amazonaws.com \
    --strip Accept --strip Accept-Encoding  --strip Connection --strip Host --strip User-Agent \
    --verbose --log-failed-requests --log-signing-process

4. Query the endpoint:

 curl http://localhost:8080
unable to proxy request - NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

So, still no luck.

[alvinlin123]

Interesting, I am not not familiar with AWS SSO, so bear with me.

I did a bit of Googling and it seems like AWS Go SDK only start to support SSO in AWS profile start at version 1.37.0.

I updated https://github.com/awslabs/aws-sigv4-proxy/pull/57 to depend on the latest version of AWS Go SDK; I also turn on the verbose logging for credential chain resolution, which should give us a bit more detail on which step went wrong during cred resolution.

When I build the image on my Mac, I had to do a go get ./ in order for the SDK update to be effective; I am not an experienced Gopher, so I am not sure if that is needed or not.

@jessemyers if you have some time, can you pull the latest change in my PR and see if it solves your problem? I am sorry about this back and forth, it is because I am not familiar with AWS SSO. But I am really keen on how to make this sigv4 proxy work for you.

If you feel like IM would accelerate the resolution to this issue, please feel free to join CNCF slack, and look for me. You should be able to find how to join CNCF Slack at: https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/attend/slack-guidelines/

[jessemyers]

@alvinlin123 I'm consistently getting errors now during the docker build:

> [build  8/10] RUN go mod download:
#14 75.38 go mod download: unrecognized import path "gopkg.in/airbrake/gobrake.v2": reading https://gopkg.in/airbrake/gobrake.v2?go-get=1: 502 Bad Gateway
#14 75.38       server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/airbrake/gobrake.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
#14 75.38 go mod download: unrecognized import path "gopkg.in/yaml.v2": reading https://gopkg.in/yaml.v2?go-get=1: 502 Bad Gateway
#14 75.38   server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[alvinlin123]

@jessemyers I built an image for you in my docker hub: https://hub.docker.com/layers/alvinlin123/aws-sigv4-proxy/issue-53/images/sha256-1d5194445bd48389c317d039ba391b9dacf3bf341636b0205b15eabf1a8bfb1f?context=explore

Give it a try when you have some time, appreciate it.

[jessemyers]

@alvinlin123 I can confirm that your latest image works with SSO login. Thank you.

[alvinlin123]

@jessemyers good news, ok so all it needs is Go SDK upgrade. I'll update my PR, get it reviewed and hopefully publish a new release in our public ECR repo.

[alvinlin123]

I have published version 1.5 to ECR public, which includes fixes in this issue. Will resolve this issue now.