raunakkapoor
commented
8 months ago cc @suniltheta
Closed raunakkapoor closed 8 months ago
raunakkapoor
commented
8 months ago cc @suniltheta
raunakkapoor
commented
8 months ago After reviewing the AWS documentation to understand signed AWS API requests and comparing it with our approach in Envoy, I identified the problem: a discrepancy in the canonical request hash.
To elaborate, I was using a Route53 record (something.com) to front the ALB, which pointed to an EKS ingress>service>pod where Envoy was running. From Envoy, I was connecting to an S3 bucket (some-bucket.s3.us-east-1.amazonaws.com). However, I missed to include the host_rewrite: some-bucket.s3.us-east-1.amazonaws.com in the typed_per_filter_config.envoy.filters.http.aws_request_signing.aws_request_signing.
When Envoy was creating the canonical request hash, it was considering the host value as something.com. In contrast, when AWS was creating the canonical request hash, it was considering the host value as some-bucket.s3.us-east-1.amazonaws.com. This discrepancy led to a signature mismatch.
The correct configuration must include host_rewrite. Without it, the hash of the canonical request will differ, resulting in a 403 error: "The request signature we calculated does not match the signature you provided".
Example of config.yaml
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: app
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: versioned-cluster
typed_per_filter_config:
envoy.filters.http.aws_request_signing:
"@type": type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigningPerRoute
aws_request_signing:
service_name: s3
region: us-west-1
use_unsigned_payload: true
host_rewrite: some-bucket.s3.us-east-1.amazonaws.com
match_excluded_headers:
- prefix: x-envoy
- prefix: x-forwarded
- exact: x-amzn-trace-id
stat_prefix: some-prefix
http_filters:
- name: envoy.filters.http.aws_request_signing
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning
service_name: s3
region: us-west-2
use_unsigned_payload: true
host_rewrite: some-bucket.s3.us-east-1.amazonaws.com
match_excluded_headers:
- prefix: x-envoy
- prefix: x-forwarded
- exact: x-amzn-trace-id
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: versioned-cluster
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: versioned-cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
AWS IAM Troubleshooting guide: https://docs.aws.amazon.com/IAM/latest/UserGuide/signature-v4-troubleshooting.html AWS documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html Envoy: https://github.com/envoyproxy/envoy/blob/v1.29.2/source/extensions/common/aws/signer_base_impl.cc
Title: Error 'SignatureDoesNotMatch' Occurred while using AWS Request Signing Filter to access an object from AWS S3
Description: I’m using Envoy debug-v1.29.2 along with the AWS request signing filter. The AWS_SIGV4 algorithm is employed to sign the requests. This setup is to access(GET call) AWS S3 objects using https routes.
Filter related configuration:
To use web identity credentials provider I have created layered_runtime and set the
envoy.reloadable_features.use_http_client_to_fetch_aws_credentials: trueCall to STS is fine and I am able to receive the temporary AWS credentials.
After that when I am accessing the route to connect to S3 bucket and access an object, it is failing with SignatureDoesNotMatch 403 error.
AWS support: I've sought assistance from AWS premium support and asked for their input. The only thing they could confirm, using backend tools to review the error, was that the error resulted from a malformed requester provided in the authentication information.
The 403 error response indicates that the correct AccessKeyID was used, which was generated after making a call to STS. In this situation, how can we identify the problem? Is there an issue occurring during the request signing process?
Any suggestions for debugging and resolving this issue would be appreciated.
[optional Relevant Links:]
Note: I have masked or mocked the security related information in the debug logs