[Feature] Enable Lambda Subscriptions to Cloudwatch Log Events

[cchan9528]

Context

We need to subscribe our lambda functions to cloud watch log events

This is for sending alerts/updates to clients during Fargate/Lambda computations

Ask

Implementation for this

[cchan9528]

This breakdown is particularly useful from the aws docs

[cchan9528]

How is this different than serverless-plugin-subscription-filter and serverless-plugin-log-subscription?

[cchan9528]

These actually do very similar things when looking into source code

serverless-plugin-log-subscription mentions AWS::Logs::LogSubscription, but this isn't a real type. Also, the source code never mentions it; it does use AWS::Logs::SubscriptionFilter and AWS:Logs::LogGroup though

serverless-plugin-subscription-filter also uses AWS::Logs::SubscriptionFilter. It does not directly use AWS::Logs::LogGroup; rather, it sets log group name when dealing with SubscriptionFilter.

They both deal with online stacks/deployments. We are trying to do offline stacks/development. Nevertheless, these two packages are very similar to what we are trying to achieve. We should consult these when developing ours.

[cchan9528]

The example in serverless-plugin-subscription-filter actually does what we're looking to do in an online way. We're trying to do it in an offline way. However, the same idea applies: under any given lambda function, we can specify which cloudwatch log event (including regex filter) should trigger it.

[cchan9528]

Looks like Serverless Framework has mechanisms to support Cloudwatch Log event-based triggers for functions (See also the 'events.cloudwatchLog' section of the "comprehensive" serverless.yml)

This is great news, because then we can use the same/parse/expect that syntax to be used when parsing options in order to run them offline 😄

[cchan9528]

How do we control the log group/stream a function's logs get sent to?

This is essential so we know which log group to monitor / trigger off of.

In other words, we know we want to trigger a lambda from a log group. But what's the log group?

It should be also in the serverless.yml file so we can share the name across functions

[cchan9528]

For what is immediately applicable for us:

• serverless naming formats
• however, the above defaults can be renamed/overriden by using the default name, and then specifying which properties you want to replace/override. Note behavior differs for different values you're trying to override - see table in link

For reference, AWS Defaults: Lambda automatically integrates with CloudWatch Logs and pushes all logs from your code to a CloudWatch Logs group associated with a Lambda function, which is named /aws/lambda/\<function name>.

[cchan9528]

So basically in this example

You can see that the function name is "write-post"

Server-less framework will give this function a log group name WriteDashPostLogGroup

- This follows from the logic in [the naming table](http://serverless.com//framework/docs-providers-aws-guide-resources)

This is important in case users try to override or customize the log group for a given function ie. Another use case

However, the general use case where they let serverless framework decide on the name should be relatively straightforward: it's just normalized lambda name + "LogGroup"

Next steps:

• figure out the normalize name function/logic:it should be the same as server less framework
• handle the general use case with normalize log group
• handle the override use case

[cchan9528]

The configuration for a lambda will look like this

1 functions:
2  myCloudWatchLog:
3    handler: myCloudWatchLog.handler
4    events:
5      - cloudwatchLog:
6          logGroup: '/aws/lambda/hello'
7          filter: '{$.userIdentity.type = Root}'

Basically, we'll want to

1. determine the log group name using the normalized function name convention

[cchan9528]

Kinesis is another way to implement common uses cases that this plugin would cover

There are sls offline plugins we can also refer to to see how they are trigerring the lambdas:

• https://github.com/emmkong/serverless-plugin-offline-kinesis-stream
  • Differences:
    • [+] Clean function invocations like fn()
    • [-] Parsing files and loading functions directly from file system instead of using functions that Serverlesss loads
    • [-] Doesn't seem like any filtering done here
• https://github.com/jacob-meacham/serverless-plugin-offline-kinesis-events
  • Differences:
    • [-] Relies on external Docker image to run

[cchan9528]

https://github.com/cchan9528/serverless-offline-cloudwatch-logs/issues/7#issuecomment-916585605

All this probably does is register the lambda with that trigger for the "oneline" mode i.e. it doesn't actually have a separate or local implementation of parsing the filter. Rather, the filter is just passed to CW subscription filters.

This is might be a blocker, so creating a new issue for a separate deep dive / investigation: https://github.com/cchan9528/serverless-offline-cloudwatch-logs/issues/8