User defined headers are dropped with aws.sign_req

[mbamber]

Short description

Opa Version: 0.63 We have seen #6456 which seems to be addressing the issue, but it doesnt seem to be working for us.

Steps To Reproduce

// my_policy.rego
get_signing_key := result {
    req := {
        "method": "post",
        "url": "https://secretsmanager.eu-west-1.amazonaws.com",
        "headers": {
            "X-Amz-Target": "secretsmanager.GetSecretValue",
            "Content-Type": "application/x-amz-json-1.1"
        },
        "body": {
            "SecretId": "mySecretId",
        }
    }
    print("req", req)
    c := {
        "aws_access_key": aws_access_key,
        "aws_secret_access_key": aws_secret_access_key,
        "aws_session_token": aws_session_token,
        "aws_service": "secretsmanager",
        "aws_region": "eu-west-1"
    }
    print("c", c)
    r := providers.aws.sign_req(req, c, now)
    print("r", r)
    resp := http.send(r)
    print("resp", resp)
    resp.status_code == 200
    result := resp.body.SecretString
}

Print statements from above

  req {"body": {"SecretId": "mySecretId"}, "headers": {"Content-Type": "application/x-amz-json-1.1", "X-Amz-Target": "secretsmanager.GetSecretValue"}, "method": "post", "url": "https://secretsmanager.eu-west-1.amazonaws.com"}
  c {"aws_access_key": "aws_access_key", "aws_region": "eu-west-1", "aws_secret_access_key": "aws_secret_access_key", "aws_service": "secretsmanager", "aws_session_token": "aws_session_token"}
  r {"body": {"SecretId": "mySecretId"}, "headers": {"Authorization": "AWS4-HMAC-SHA256 Credential=aws_access_key/19700101/eu-west-1/secretsmanager/aws4_request,SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target,Signature=5c40341a1f6ee94091c9a8653cc059cc84bfd8713c0ad553da8b0f29b0769a4a", "host": "secretsmanager.eu-west-1.amazonaws.com", "x-amz-date": "19700101T002835Z", "x-amz-security-token": "aws_session_token"}, "method": "post", "url": "https://secretsmanager.eu-west-1.amazonaws.com"}

Expected behavior

We believe the signature is being calculated correctly, but we would expect r to show the headers that we can see in req.

Additional context

[ashutosh-narkar]

@c2zwdjnlcg is this something you can look into? @mbamber if you'd like to submit a fix, feel free to do so. Thanks.

[mbamber]

If I knew how to fix it, I would have already submitted a PR 😛

I've spent the last few hours trying to workout how all the tests etc work but I must admit I'm somewhat floundering. If you can point me in the direction of how to debug the tests, that would certainly help.

[mbamber]

I did manage to discover the following though:

Given the following opa test which uses the same data from test/cases/testdata/providers-aws/aws-sign_req.yaml

test_headers if {
    req := {"method": "get", "url": "http://example.com", "headers": {"foo": "bar"}}
    expected := {
      "headers": {
        "Authorization": "AWS4-HMAC-SHA256 Credential=MYAWSACCESSKEYGOESHERE/20151228/us-east-1/s3/aws4_request,SignedHeaders=foo;host;x-amz-content-sha256;x-amz-date,Signature=8f1dc7c9b9978356a0d0989fd26a95307f4f8a4aa264d8220647b7097d839952",
        "host": "example.com",
        "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
        "x-amz-date": "20151228T140825Z",
        "foo": "bar"
      },
      "method": "get",
      "url": "http://example.com"
    }
    aws_config := {
      "aws_access_key": "MYAWSACCESSKEYGOESHERE",
      "aws_secret_access_key": "MYAWSSECRETACCESSKEYGOESHERE",
      "aws_service": "s3",
      "aws_region": "us-east-1",
    }
    r := providers.aws.sign_req(req, aws_config, 1451311705000000000)
    print(r)
    print(expected)
    r == expected
}

... I get the following output:

[redacted].test_headers: FAIL (447.645µs)

  {"headers": {"Authorization": "AWS4-HMAC-SHA256 Credential=MYAWSACCESSKEYGOESHERE/20151228/us-east-1/s3/aws4_request,SignedHeaders=foo;host;x-amz-content-sha256;x-amz-date,Signature=8f1dc7c9b9978356a0d0989fd26a95307f4f8a4aa264d8220647b7097d839952", "host": "example.com", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20151228T140825Z"}, "method": "get", "url": "http://example.com"}
  {"headers": {"Authorization": "AWS4-HMAC-SHA256 Credential=MYAWSACCESSKEYGOESHERE/20151228/us-east-1/s3/aws4_request,SignedHeaders=foo;host;x-amz-content-sha256;x-amz-date,Signature=8f1dc7c9b9978356a0d0989fd26a95307f4f8a4aa264d8220647b7097d839952", "foo": "bar", "host": "example.com", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20151228T140825Z"}, "method": "get", "url": "http://example.com"}

which I think confirms my suspicion that this is a bug (either that or I'm misunderstanding the use of this builtin)

[c2zwdjnlcg]

I can't reproduce, you sure you have the right version of the binary? what's the full output of opa version

[mbamber]

You're absolutely right, sorry for wasting your time @c2zwdjnlcg

We were using a out of date version in the tests and once upgraded we can confirm that headers are not dropped