Verifying the signatures of Amazon SNS messages

[mmattioli]

The official documentation regarding verifying SNS signatures clearly states:

When possible, use one of the supported AWS SDKs for Amazon SNS to validate and verify messages. For example, with the AWS SDK for PHP you would use the isValid method from the MessageValidator class.

I know some have made their own and published them on GitHub but they're not maintained well and I don't see why this isn't something included in the Python SDK.

Does anyone have any suggestion or guidance on how to simply verify incoming SNS messages?

[mmattioli]

Here's my quick and dirty solution based on the documentation Amazon provides; it requires M2Crypto and Requests. sns_payload is the payload that SNS posts to an HTTP/s endpoint parsed in JSON.

import base64
import requests
from M2Crypto import X509

def valid_sns_message(sns_payload):

    # Can only be one of these types.
    if sns_payload["Type"] not in ["SubscriptionConfirmation", "Notification", "UnsubscribeConfirmation"]:
        return false

    # Amazon SNS currently supports signature version 1.
    if sns_payload["SignatureVersion"] != "1":
        return false

    # Fields for a standard notification.
    fields = ["Message", "MessageId", "Subject", "Timestamp", "TopicArn", "Type"]

    # Fields for subscribe or unsubscribe.
    if sns_payload["Type"] in ["SubscriptionConfirmation", "UnsubscribeConfirmation"]
        fields = ["Message", "MessageId", "SubscribeURL", "Timestamp", "Token", "TopicArn", "Type"]

    # Build the string to be signed.
    string_to_sign = ""
    for field in fields:
        string_to_sign += field + "\n" + sns_payload[field] + "\n"

    # Decode the signature from base64.
    decoded_signature = base64.b64decode(sns_payload["Signature"])

    # Retrieve the certificate.
    certificate = X509.load_cert_string(requests.get(sns_payload["SigningCertURL"]).text)

    # Extract the public key.
    public_key = certificate.get_pubkey()

    # Amazon SNS uses SHA1withRSA.
    # http://sns-public-resources.s3.amazonaws.com/SNS_Message_Signing_Release_Note_Jan_25_2011.pdf
    public_key.reset_context(md = "sha1")
    public_key.verify_init()

    # Sign the string.
    public_key.verify_update(string_to_sign.encode())

    # Verify the signature matches.
    verification_result = public_key.verify_final(decoded_signature)

    # M2Crypto uses EVP_VerifyFinal() from openssl as the underlying verification function.
    # 1 indicates success, anything else is either a failure or an error.
    if verification_result != 1:
        return False

    return True

[swetashre]

@mmattioli - Thank you for your post and thank you for providing your solution. Currently this is not supported in python sdk so i will mark this as feature request. You can find some more information in this issue https://github.com/boto/boto3/issues/1469.

[wlwg]

I hope this feature can be included in the official python SDK soon too! Before that, I'll just keep using my own implementation: https://github.com/wlwg/aws-sns-message-validator

[hinnerk]

The code above is not valid Python code, bona-fide seems unmaintained since 2015, aws-sns-message-validator does not install because of outdated dependencies, aws-sns-message-validator2 looks like a very temporary upload of a somewhat patched aws-sns-message-validator, validate_aws_sns_message seems outdated and sns-message-validator just seems to be another copy of aws-sns-message-validator.

I do completely understand that the authors of these modules chose to not invest their time in maintenance and documentation; for small projects these are thankless jobs and making sure that something with crypto is always up to date with the latest security patches just plain sucks.

But in consequence there's no production ready solution for SNS signature validation. We all know what that means: Most Python production code runs without SNS signature validation. Nobody invests a week for an optional security component and pays the recurring maintenance costs. See above for plenty examples. After all, features are working without signature validation, are they not?

So please make the Python world a safer place by integrating and maintaining SNS message validation.

[mmattioli]

@hinnerk when you say "the code above" are you referring to aws-sns-message-validator or the code in my comment?

[hinnerk]

@mmattioli the code in your comment, the import for base64 in line 27 is missing and lines 8 and 12 (false) will raise a NameError if reached. I'm sorry that my language wasn't precise, the code is not invalid, but it will raise unexpectedly.

[mmattioli]

@hinnerk fixed 😁

[stebunovd]

Here's an adapted version of the code above that we use in production for Teamplify - https://gist.github.com/stebunovd/c4122c5a9ae70185c20c7b2f1ec03cfc

This is mostly an original version from @mmattioli (thank you for your work!) with minor fixes, re-written in a more defensive way, and with a basic cache for signing certificates (so that it doesn't make a new HTTP request for each signature validation).

Hopefully, one day this feature will be included in boto.

[DataGhost]

I mean this is nice and all, and no offense meant @mmattioli but you might as well skip the entire validation if this is what you're using. The certificate used to validate the message is user-supplied and not checked if it's an actual Amazon certificate or not, so anyone can construct an arbitrary message that passes validation. The code by @stebunovd ~looks better already~ does not look much better in terms of security, since the only validation is that the certificate URL is hosted on amazonaws.com, which is the case for all S3 buckets. I guess this is one extra reason why this should be in the SDK.

[stebunovd]

hey @DataGhost, good point about cert validation. How would you implement such a check?

[DataGhost]

Well, I don't know yet (edit: I guess I do after writing this comment), I haven't yet read all documentation on this subject. I have read a fair deal more now while writing this comment and gotten slightly annoyed about the way they set this up and the accompanying documentation, but it seems workable if you piece the right resources together.

The only "real" documentation I've been able to find here is http://sns-public-resources.s3.amazonaws.com/SNS_Message_Signing_Release_Note_Jan_25_2011.pdf and the text mentions the certificates being served from "amazonaws.com" as the sole criterion that should be checked. I wholeheartedly disagree with only that as documentation. It's even worse considering I've found that document only through this bug report and it seems completely absent on the current, official documentation https://docs.aws.amazon.com/sns/latest/dg/sns-verify-signature-of-message.html. They might as well have just included a checksum for every message because that's basically all it tells you to do now. Even though that documentation explicitly lists some steps to help prevent spoofing attacks, they're very inadequately documented and at least one of the steps has not been made easy to perform at all. So that definitely needs improvement.

Anyway, going by the PDF (not the HTML) they do list some explicit URLs as the sole URLs from which the signing certificates should be available, so that can be enforced with a check that's much stricter than just "amazonaws.com". Thing is, the certificates are served over http (at least in that document) and there is no easily-accessible chain of trust to verify them against by default, so if you have enough tin foil, this is not secure at all. Luckily, the certificates are available through https as well so I suggest both checking the URL format and forcing the fetch over https at the very least. This is also suggested in the HTML docs, and they're also already sending out https URLs (as per the normal docs) so that's a good thing. It still must be checked, though!

So for validation, I'd probably go with an URL validator for:

• ://sns.[region].amazonaws.com/SimpleNotificationService.pem
• ://sns.[region].amazonaws.com/SimpleNotificationService-[certSerialNumberInHex].pem while forcing https and making absolutely sure "[region]" does not contain any dots so it can't accidentally be served from some user's S3 bucket.

The URL filtering and forced HTTPS together should in my opinion be secure enough for most use cases and should rule out tampering with the certificate by an attacker, since it's served over a supposedly secure channel. You can probably stop reading here. There's still the option of an attacker that managed to obtain a valid certificate for sns.[region].amazonaws.com and serving their own SNS certificate as MITM but you might have bigger problems if that's the case.

This still leaves the suggested step of validating the certificate itself, unfortunately that's not documented and doesn't seem super-straightforward. I mean, it's easy to do, if you have all the pieces, but that isn't the case. Looking at one of the certificates, it's got sns.amazonaws.com as CN and is signed by CN=Amazon, OU=Server CA 1B. That's not by their top-level certificate which is included in my OS' default certificate store but by an intermediate which isn't, and isn't supplied with the certificate nor is it documented where to get it from. I found it on https://www.amazontrust.com/repository/ under "Other CAs" and it turns out they have a bunch of intermediates, not just the one, any of which could potentially be used to sign future certificates. With the intermediates only, it should be possible to validate the certificate against Amazon's Root Certificate which should already be present on your system. Having to fetch the intermediates from there seems like a less-than-ideal situation though, but it could work. That'd allow you to actually establish the chain of trust on the provided certificate itself, but this is probably overkill and not specifically any more secure because the same unlikely attack vector of a forged certificate for amazontrust.com exists. It would be nice though if the used intermediate was easily supplied along with the SimpleNotificationService.pem, that would have made certificate validation trivial.

In addition to the URL check and HTTPS fetch I meantioned above, when not doing full certificate validation I could personally consider doing a simple string check on the CN of the certificate. Thing is, they haven't explicitly documented that it's always going to be CN=sns.amazonaws.com for the signing cert, so if that changes for any reason, it'll break validation.

Again, vastly preferring a singular implementation of all the required security in the SDK where it can be analysed by others rather than everyone having to implement their own, potentially incomplete, security measures.

[stebunovd]

Wow, this is a really deep answer, thanks! I've updated the gist, so now it includes a more strict check for the certificate URL and also checks for the CN of the certificate.

[jcpage]

FYI, Amazon has some sample code in their knowledge center, but's it's no better in terms of cert checking than anything else...

https://aws.amazon.com/premiumsupport/knowledge-center/sns-verify-message-authenticity/

Edit: here's a decent example I found doing it in Javascript (ie it does check the URL, but doesn't do any other validation), including retries and caching of the cert (which seems essential for a production app):

https://cloudonaut.io/verify-sns-messages-delivered-via-http-or-https-in-node-js/

Agreed that no matter which SDK, this could be a core feature for security's sake!

[BarneyPlummer]

Thanks for the gist @stebunovd - a couple of issues that I've run into with it are that:

• Notification messages might not contain a Subject (see https://docs.aws.amazon.com/sns/latest/dg/sns-verify-signature-of-message.html), in which case the Subject is not used to build up the string to sign
• SNS notifications sent to Lambda contain a SigningCertUrl rather than a SigningCertURL (ugh... - see https://aws.amazon.com/blogs/mobile/invoking-aws-lambda-functions-via-amazon-sns/)

More reason for it to be part of boto!

[judgeaxl]

Wouldn't one also have to verify that the TopicARN matches an expected ARN or ARN pattern, or at the very least your own account(s)? Otherwise, what prevents me from setting up another SNS topic and have it send messages to an HTTPS endpoint I've "discovered"? IIRC the endpoint must be public for SNS to work. This of course requires that the endpoint auto-subscribes, but I imagine a lot of them do for convenience.

[jimmoffet]

Found this discussion enlightening. Following the thread, I've updated @stebunovd's implementation to address the comments by @BarneyPlummer and @judgeaxl above. The gist is here. I use redis for caching across workers, added it as optional. Also, added optional ARN validation. I'm only checking for expected SNS topic names, but if you name them something like myapp_myevent_highentropystring, they'd be both human-readable and unguessable.

+n for adding it to boto!

[gabelton]

Thanks for creating that gist @stebunovd . I've not worked much with crypto libraries, but noticed that the original m2crypto library appears to have been archived, although eventbrite seem to be maintaining a fork. Do you have a sense of whether it would be possible to substitute cryptography instead of m2? I've followed their x509 instructions, but getting confused once I get to this part of the code

 # Amazon SNS uses SHA1withRSA.
 # http://sns-public-resources.s3.amazonaws.com/SNS_Message_Signing_Release_Note_Jan_25_2011.pdf
 public_key.reset_context(md='sha1')
 public_key.verify_init()

 # Sign the string.
 public_key.verify_update(string_to_sign.encode())

 # Verify the signature matches.
 verification_result = public_key.verify_final(decoded_signature)

While I'm here, +1 obvs for adding this to boto itself. Anyone know of any plans to include this in an upcoming release cycle?

[bentsku]

Hello @gabelton,

Maybe this can help, I've implemented it using cryptography too. Disclaimer: this does not take into account any safety like talked earlier in this thread, this is just for the replacement of m2crypto by cryptography.

Starting at the point as your snippet:

# the import used:
from cryptography import x509
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding

cert_url = message["SigningCertURL"]
get_cert_req = requests.get(cert_url)

cert = x509.load_pem_x509_certificate(get_cert_req.content)

message_signature = message["Signature"]
# decode the signature from base64.
decoded_signature = base64.b64decode(message_signature)

message_sig_version = message["SignatureVersion"]
signature_hash = hashes.SHA1() if message_sig_version == "1" else hashes.SHA256()

# verify the signature value with cert, if the signature is not valid, it will raise `InvalidSignature`
cert.public_key().verify(
    decoded_signature,
    string_to_sign.encode(),
    padding=padding.PKCS1v15(),
    algorithm=signature_hash,
)

Hope this can help!

Note: for the padding used to verify, I've taken it from this StackOverflow thread where they looked at the padding used in the C# SDK util to verify SNS messages: https://stackoverflow.com/a/77654997

[hiven]

This should be in boto??