InvalidSignatureException for GetSamplingRules when using daemon in EKS with Istio service mesh

[SPopenko]

Hello guys! Did someone faced with the issue described below and have any recommendations how to avoid that?

Environment:

• AWS EKS
• Istio Service mesh
• Side cars are deployed only in default namespace with product services
• Daemon is deployed in separate namespace, that doesn't add istio side cars

Problem: Istio side car (envoy) may add additional headers to each outbound request when any service calls X-ray daemon to get GetSamplingRules and that cause InvalidSignatureException from daemon.

Here is trace: From busy box with side car

` /home # curl -X POST 10.0.2.88:2000/GetSamplingRules -v

• Trying 10.0.2.88:2000...
• Connected to 10.0.2.88 (10.0.2.88) port 2000 (#0)

  POST /GetSamplingRules HTTP/1.1 Host: 10.0.2.88:2000 User-Agent: curl/7.81.0 Accept: /

• Mark bundle as not supporting multiuse < HTTP/1.1 403 Forbidden < content-length: 192 < content-type: application/json < date: Fri, 14 Apr 2023 09:02:34 GMT < strict-transport-security: max-age=31540000; includeSubDomains < x-amzn-errortype: InvalidSignatureException < x-amzn-requestid: 4d4f98f2-9230-45c7-9b2c-314f94d44275 < x-content-type-options: nosniff < x-envoy-upstream-service-time: 34 < server: envoy <
• Connection #0 to host 10.0.2.88 left intact {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}/home #`

From busy box without side car `bash-5.2$ curl -X POST 10.0.2.88:2000/GetSamplingRules -v

• Trying 10.0.2.88:2000...
• Connected to 10.0.2.88 (10.0.2.88) port 2000 (#0)

  POST /GetSamplingRules HTTP/1.1 Host: 10.0.2.88:2000 User-Agent: curl/7.88.1 Accept: /

  < HTTP/1.1 200 OK < Content-Length: 727 < Content-Type: application/json < Date: Fri, 14 Apr 2023 09:04:41 GMT < Strict-Transport-Security: max-age=31540000; includeSubDomains < X-Amzn-Requestid: 12c7f891-7fa4-4373-b94a-e7b5aed48a31 < X-Content-Type-Options: nosniff <

• Connection #0 to host 10.0.2.88 left intact {"NextToken":null,"SamplingRuleRecords":[{"CreatedAt":1.681386086E9,"ModifiedAt":1.681386427E9,"SamplingRule":{"Attributes":{},"FixedRate":0.0,"HTTPMethod":"","Host":"","Priority":1,"ReservoirSize":0,"ResourceARN":"","RuleARN":"arn:aws:xray:us-east-1:473843136979:sampling-rule/Skip-health-requests","RuleName":"Skip-health-requests","ServiceName":"","ServiceType":"","URLPath":"/health","Version":1}},{"CreatedAt":0.0,"ModifiedAt":0.0,"SamplingRule":{"Attributes":{},"FixedRate":0.05,"HTTPMethod":"","Host":"","Priority":10000,"ReservoirSize":1,"ResourceARN":"","RuleARN":"arn:aws:xray:us-east-1:473843136979:sampling-rule/Default","RuleName":"Default","ServiceName":"","ServiceType":"","URLPath":"*","Version":1}}]}bash-5.2$ `

[wangzlei]

Provide XRay SDK GetSamplingTargets/GetSamplingRules mechanism for your reference: GetSamplingTargets/GetSamplingRules request is sent from xray SDK(running in user's application) in plain http, the request arrives at xray daemon, then is signed and forward to xray service.

XRay SDK  -> XRay Daemon(sigv4) -> XRay service

So updating headers before XRay Daemon is ok. But update the request after XRay daemon will cause invalid exception.

[SPopenko]

@wangzlei you are right, but the issue was a bit deeper.

Envoy proxy (that is used as Istio sidecar) makes modifications for all outbound requests from Xray SDK (in own service) to Xray Daemon. It adds bunch of headers but only one cause the issue.

Envoy adds "X-Forwarded-Proto: http" header, that is fair enough, since like you said, SDK and daemon communicate via plain http. This header cause invalid signature exception from AWS X-ray service if included in request (even if one run it from postman).

My workaround is adding custom Lua filter for envoy:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: xray-patch
  namespace: default
  annotations:
    name: Fix commnuication between service and xray 
    description: Replase x-forwarded-proto to https
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          portNumber: 2000
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua
          typed_config:
            '@type': 'type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua'
            inlineCode: |
              function envoy_on_request(request_handle)
                request_handle:headers():replace("x-forwarded-proto", "https")
              end

Note: envoy filters are pretty 'dangerous' (could break your mesh if works in a wrong way) and are very sensitive to Istio/Envoy version