Using a single lambda function for multiple SNS triggers

[gjwilson21]

Hi,

We have a use case for lambda function to send some metrics about files being uploaded to S3 to a monitoring server. There are multiple SNS topics which receives events from S3 buckets. For example, TopicA receives notification for S3 when a file is created under BucketA/folder1 and TopicB receives notification when a file is created under BucketB/folder1, etc. The lambda function that we would like to create is a generic function which can read the message from the event and take appropriate actions. Is there a way to have a single lambda function subscribe to multiple SNS topics?

Right now, I can only think of the below approach, which deploys multiple lambda functions(one per sns topic) @app.on_sns_message(topic='SNSTest1') def handle_SNSTest(event): process_sns_message(event)

@app.on_sns_message(topic='SNSTest2') def handle_SNSTest2(event): process_sns_message(event)

@app.on_sns_message(topic='SNSTest3') def handle_SNSTest3(event): process_sns_message(event)

Is there a better way to write this?

Thanks

[stealthycoin]

I think that's the only way to do it as of now. Whats the drive for only having one lambda function?

[gjwilson21]

Thanks for your response @stealthycoin . The reason for having one lambda function is, we have close to 50 SNS topics which gets s3 event notifications, and the logic implemented in lambda is common fo all the topics. So instead of maintaining 50 separate lambda functions, we can maintain a single lambda function.

Since there is no way to implement a single lambda function which can listen to multiple sns topics using Chalice, I am planning to subscribe a sqs topic to those 50+ SNS topics, and use the sqs as a trigger for the lambda function. Do you know of any disadvantages to this approach?

[stealthycoin]

I can mark this as a feature request. But in terms of how I would implement that I would just have 1 SNS topic. My consumers would then just filter the messages for the bucket they are interested in. Whats the reason you have 50 SNS topics?

[gjwilson21]

The reason is, we already have the SNS topics which are specific to certain folders in S3 bucket and there are customers who are subscribed to those topics and some of the topics are exclusive to a group of customers so we cannot send all the messages to every customer.

And as you might already know, we can only set one s3 event notification per unique prefix, so we cannot have the messages sent to a new topic in addition to the topics that were already setup.s

Thanks for your response!

[dmulter]

Another case to consider for @stealthycoin is with a very large number of event messages being posted to a single SNS topic, you potentially have way more Lambda invocations than are normally necessary for a consumer only interested in specific events. Ideally you could stack decorators for SNS/SQS events. It would also help to have underlying API control without a decorator so you could dynamically connect Lambda functions to event sources. Not sure if this is possible now.