Attachments are incompatible with S3 Pre-signed URLs

[stv0g]

I've been trying to deliver notifications using pre-signed URLs for attachments stored in S3.

This is broken as Apprise will mangle the query-string of the URL before fetching it from the S3 server. Hence the signature is invalid and the GET request fails.

Solution:

Apprise shall not mangle the URL when fetching remote attachments

[caronc]

Can you provide an example? I don't know what is being mangled.

[stv0g]

Sure, here is the relevant part of the log:

09:12:08.764 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): localhost:9000
09:12:08.774 DEBUG urllib3.connectionpool https://localhost:9000 "GET /seguro/attachments/1bb5d4b5-acc7-4b87-98b9-5aadeb1b4c0f/test.txt?x-amz-security-token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJDQjFYODBLVEhEUlIwSEFYRTlHNiIsImF1ZCI6WyJPUEFMLVJUIEdlcm1hbnkgR21iSCJdLCJleHAiOjE3MTQxMTkxMjMsImlzcyI6IlNFR3VSbyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkiLCJwYXJlbnQiOiJ0bHM6YWRtaW4iLCJzdWIiOiJhZG1pbiJ9.uIfEn8EKi1OCYl5xDBxUgz13-BBBAdIZET6cHalvC3lacX2aA7fbARA0dS5-pI2ux6573OAjr7Ki9Ky8WJWhYw&x-amz-algorithm=AWS4-HMAC-SHA256&x-amz-credential=CB1X80KTHDRR0HAXE9G6%2F20240426%2Fminio%2Fs3%2Faws4_request&x-amz-date=20240426T071208Z&x-amz-expires=604800&x-amz-signedheaders=host&x-amz-signature=c55f84cd6f483d5a937297d4428a78fbfeb4c0fbfbc33f5362bc66e1dcca3f5b HTTP/1.1" 403 417
09:12:08.775 ERROR apprise A Connection error occurred retrieving HTTP configuration from localhost.
09:12:08.775 DEBUG apprise Socket Exception: 403 Client Error: Forbidden for url: https://localhost:9000/seguro/attachments/1bb5d4b5-acc7-4b87-98b9-5aadeb1b4c0f/test.txt?x-amz-security-token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJDQjFYODBLVEhEUlIwSEFYRTlHNiIsImF1ZCI6WyJPUEFMLVJUIEdlcm1hbnkgR21iSCJdLCJleHAiOjE3MTQxMTkxMjMsImlzcyI6IlNFR3VSbyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkiLCJwYXJlbnQiOiJ0bHM6YWRtaW4iLCJzdWIiOiJhZG1pbiJ9.uIfEn8EKi1OCYl5xDBxUgz13-BBBAdIZET6cHalvC3lacX2aA7fbARA0dS5-pI2ux6573OAjr7Ki9Ky8WJWhYw&x-amz-algorithm=AWS4-HMAC-SHA256&x-amz-credential=CB1X80KTHDRR0HAXE9G6%2F20240426%2Fminio%2Fs3%2Faws4_request&x-amz-date=20240426T071208Z&x-amz-expires=604800&x-amz-signedheaders=host&x-amz-signature=c55f84cd6f483d5a937297d4428a78fbfeb4c0fbfbc33f5362bc66e1dcca3f5b
09:12:08.775 ERROR apprise Could not access attachment https://localhost:9000/seguro/attachments/1bb5d4b5-acc7-4b87-98b9-5aadeb1b4c0f/test.txt?rto=4.0&cto=4.0&verify=yes&cache=yes&x-amz-security-token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJDQjFYODBLVEhEUlIwSEFYRTlHNiIsImF1ZCI6WyJPUEFMLVJUIEdlcm1hbnkgR21iSCJdLCJleHAiOjE3MTQxMTkxMjMsImlzcyI6IlNFR3VSbyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkiLCJwYXJlbnQiOiJ0bHM6YWRtaW4iLCJzdWIiOiJhZG1pbiJ9.uIfEn8EKi1OCYl5xDBxUgz13-BBBAdIZET6cHalvC3lacX2aA7fbARA0dS5-pI2ux6573OAjr7Ki9Ky8WJWhYw&x-amz-algorithm=AWS4-HMAC-SHA256&x-amz-credential=CB1X80KTHDRR0HAXE9G6%2F20240426%2Fminio%2Fs3%2Faws4_request&x-amz-date=20240426T071208Z&x-amz-expires=604800&x-amz-signedheaders=host&x-amz-signature=c55f84cd6f483d5a937297d4428a78fbfeb4c0fbfbc33f5362bc66e1dcca3f5b.
09:12:08.775 DEBUG apprise HTTP Attachment Fetch URL: https://localhost:9000/seguro/attachments/1bb5d4b5-acc7-4b87-98b9-5aadeb1b4c0f/test.txt (cert_verify=True)
09:12:08.776 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): localhost:9000
09:12:08.785 DEBUG urllib3.connectionpool https://localhost:9000 "GET /seguro/attachments/1bb5d4b5-acc7-4b87-98b9-5aadeb1b4c0f/test.txt?x-amz-security-token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJDQjFYODBLVEhEUlIwSEFYRTlHNiIsImF1ZCI6WyJPUEFMLVJUIEdlcm1hbnkgR21iSCJdLCJleHAiOjE3MTQxMTkxMjMsImlzcyI6IlNFR3VSbyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkiLCJwYXJlbnQiOiJ0bHM6YWRtaW4iLCJzdWIiOiJhZG1pbiJ9.uIfEn8EKi1OCYl5xDBxUgz13-BBBAdIZET6cHalvC3lacX2aA7fbARA0dS5-pI2ux6573OAjr7Ki9Ky8WJWhYw&x-amz-algorithm=AWS4-HMAC-SHA256&x-amz-credential=CB1X80KTHDRR0HAXE9G6%2F20240426%2Fminio%2Fs3%2Faws4_request&x-amz-date=20240426T071208Z&x-amz-expires=604800&x-amz-signedheaders=host&x-amz-signature=c55f84cd6f483d5a937297d4428a78fbfeb4c0fbfbc33f5362bc66e1dcca3f5b HTTP/1.1" 403 417

Which shows that the request is made against:

https://localhost:9000/seguro/attachments/1bb5d4b5-acc7-4b87-98b9-5aadeb1b4c0f/test.txt?x-amz-security-token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJDQjFYODBLVEhEUlIwSEFYRTlHNiIsImF1ZCI6WyJPUEFMLVJUIEdlcm1hbnkgR21iSCJdLCJleHAiOjE3MTQxMTkxMjMsImlzcyI6IlNFR3VSbyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkiLCJwYXJlbnQiOiJ0bHM6YWRtaW4iLCJzdWIiOiJhZG1pbiJ9.uIfEn8EKi1OCYl5xDBxUgz13-BBBAdIZET6cHalvC3lacX2aA7fbARA0dS5-pI2ux6573OAjr7Ki9Ky8WJWhYw&x-amz-algorithm=AWS4-HMAC-SHA256&x-amz-credential=CB1X80KTHDRR0HAXE9G6%2F20240426%2Fminio%2Fs3%2Faws4_request&x-amz-date=20240426T071208Z&x-amz-expires=604800&x-amz-signedheaders=host&x-amz-signature=c55f84cd6f483d5a937297d4428a78fbfeb4c0fbfbc33f5362bc66e1dcca3f5b

While the URL I requested was:

https://localhost:9000/seguro/attachments/1bb5d4b5-acc7-4b87-98b9-5aadeb1b4c0f/test.txt?X-Amz-Security-Token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJDQjFYODBLVEhEUlIwSEFYRTlHNiIsImF1ZCI6WyJPUEFMLVJUIEdlcm1hbnkgR21iSCJdLCJleHAiOjE3MTQxMTkxMjMsImlzcyI6IlNFR3VSbyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkiLCJwYXJlbnQiOiJ0bHM6YWRtaW4iLCJzdWIiOiJhZG1pbiJ9.uIfEn8EKi1OCYl5xDBxUgz13-BBBAdIZET6cHalvC3lacX2aA7fbARA0dS5-pI2ux6573OAjr7Ki9Ky8WJWhYw&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=CB1X80KTHDRR0HAXE9G6%2F20240426%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20240426T071208Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=c55f84cd6f483d5a937297d4428a78fbfeb4c0fbfbc33f5362bc66e1dcca3f5b

The main difference I can sport here is the capitalization of the query string arguments.

[stv0g]

I guess this is the culprint:

https://github.com/caronc/apprise/blob/08cb018e11bdc2f21ac9ad409484959f78a09f5a/apprise/utils.py#L611C1-L612C1

[caronc]

Got it, this one I'll need to think about this one. I do see what you're talking about though, thank you for pinpointing it so quickly.

[caronc]

Would this work if you use curl instead? I thought the x-amz-security-token would be a header entry, not something on the URL?

If you need to make a query and set the Header, you can alter the kwarg slightly:

# Basically Apprise sets everything to be on the GET URL UNLESS you prefix the entry with a plus '+'
# +key=value will actually get pushed upstream as a header entry
attach=http://localhost:9000/?+X-Amz-Security-Token=token&regularkw=value&+X-Another-Header=Value

Perhaps this is what you're trying to achive?

[stv0g]

For GET requests they should be passed as query string parameters:

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

But I think the fact that Apprise's parse_qsd() function is converting all query string argument keys to lowercase is violating the standard:

See: URI RFC 3986 Section 6.2.2.1

6.2.2.1. Case Normalization

For all URIs, the hexadecimal digits within a percent-encoding triplet (e.g., "%3a" versus "%3A") are case-insensitive and therefore should be normalized to use uppercase letters for the digits A-F.

When a URI uses components of the generic syntax, the component syntax equivalence rules always apply; namely, that the scheme and host are case-insensitive and therefore should be normalized to lowercase. For example, the URI HTTP://www.EXAMPLE.com/ is equivalent to http://www.example.com/. The other generic syntax components are assumed to be case-sensitive unless specifically defined otherwise by the scheme (see Section 6.2.3).

I was also unable to find any HTTP(s) scheme specific URI normalization rules.

I am currently using the following work-around by sub-classing AttachHTTP:

from apprise.attachment.AttachHTTP import AttachHTTP

class FixedAttachHTTP(AttachHTTP):

    def __init__(self, url):
        super().__init__(**AppriseAttachHTTP.parse_url(url))

        pr = urllib.parse.urlparse(url)
        qs = urllib.parse.parse_qs(pr.query)

        self.qsd = {k: v[0] for k, v in qs.items()}

[stv0g]

Sorry I missed this question:

Would this work if you use curl instead? I thought the x-amz-security-token would be a header entry, not something on the URL?

Yes, taking my original URL from my comment above (https://github.com/caronc/apprise/issues/1118#issuecomment-2078775452) works via CURL and also plain urllib3.request().

Also using my FixedAttachHTTP class from my previous comment fixes the issue.

[caronc]

If you could have a look at the attached PR (there are instructions on how to use it) and let me know if that works for you?

[caronc]

Code merged; will appear in next release - closing ticket