AWS4Auth produces incomplete canonical query string

[thurse93]

Hi!

I'm experiencing a problem that AWS4 authentication does not seem to work correctly when communicating with the Ceph RADOS Gateway Admin Operations API (https://docs.ceph.com/en/latest/radosgw/adminops/#admin-operations) which uses AWS Signature Version 4 to authenticate. Given the following snippets:

import requests
from requests_aws4auth import AWS4Auth

auth = AWS4Auth("access_key", "secret",
                "default", "s3")

with requests.Session() as client:
    response = client.put("https://ceph-host/admin/user", auth=auth,
                          params={"format": "json", "uid": "testtenant$testuser",
                                  "display-name": "Some Test Display name."})

import httpx
from httpx_auth import AWS4Auth

auth = AWS4Auth(access_id="access_key", secret_key="secret",
                region="default", service="s3")

with httpx.Client() as client:
    response = client.put("https://ceph-host/admin/user", auth=auth,
                                 params={"format": "json", "uid": "testtenant$testuser",
                                         "display-name": "Some Test Display name."})

The first one works like a charm, but the second one using httpx and httpx_auth does not (RADOS Gateway responds with a HTTP status code of 403 an a "SignatureDoesNotMatch" error).

(I chose requests_aws4auth as a counter example here, because it's getting used internally by one of the python libraries the Ceph Admin Ops documentation suggests using: https://github.com/UMIACS/rgwadmin)

It seems that the reason behind this behaviour is that the implementation of httpx_auth does not expect query parameters with arbitrary spaces in them:

    def _amz_cano_querystring(qs: str) -> str:
        """
        Parse and format querystring as per AWS4 auth requirements.
        Perform percent quoting as needed.
        qs -- querystring
        """
        safe_qs_amz_chars = "&=+"
        safe_qs_unresvd = "-_.~"
        qs = unquote(qs)   # 'Some%20Test%20Display%20name.' gets unquoted here, so split() produces an incorrect result
        space = " "
        qs = qs.split(space)[0]
        qs = quote(qs, safe=safe_qs_amz_chars)
        qs_items = {}
        for name, vals in parse_qs(qs, keep_blank_values=True).items():
            name = quote(name, safe=safe_qs_unresvd)
            vals = [quote(val, safe=safe_qs_unresvd) for val in vals]
            qs_items[name] = vals
        qs_strings = []
        for name, vals in qs_items.items():
            for val in vals:
                qs_strings.append("=".join([name, val]))
        qs = "&".join(sorted(qs_strings))
        return qs

I'm not very familiar with the AWS Signature Spec, so I can't tell whether this is intended behaviour or not.

If it is, maybe the function should raise something if it detects more than one space in the query string. If it isn't, and it should work with other object storage provider APIs such as Ceph, maybe one could change the implementation to act a bit more forgiving as in requests_aws4auth.

[Colin-b]

Hello @thurse93

Thanks for reporting this issue. Can you try with version 0.18.0 to see if it works? And then try again with 0.19.0 as it seems related to this change.

[thurse93]

Unfortunately not, it's still the same issue since the value of the qs parameter didn't change.

I'm trying to understand the querystring logic by comparing it to the documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

But I'm too blind to see where this space-based split comes from. Can you explain whats the reason behind that?

[Colin-b]

Hello @thurse93 , I think the issue comes from the fact that requests-aws4auth made some changes since we ported this module. We would need to look over the modification that occurred, compare them with the AWS documentation and also find out what are the appropriate test cases to write to ensure non regression on those features/bug fixes.

[Colin-b]

Hello @thurse93

Release 0.20.0 is now available on pypi and should fix this issue (amongst others). I reviewed the code from requests-aws4auth and their test suite, and compared the whole to the AWS documentation. After consideration I decided to get a code base closer to AWS documentation than requests-aws4auth.

If you notice any issue most likely due to undocumented behavior on AWS side, please do not hesitate to raise an issue.

Thanks again