techservicesillinois / terraform-aws-cloudwatch-log-group

Provide CloudWatch log group with default forwarding to Splunk
MIT License
3 stars 3 forks source link

Change default for disable_subscription_filter? #11

Open JonRoma opened 3 years ago

JonRoma commented 3 years ago

The disable_subscription_filter currently defaults to false, which automatically subscribes the log group to the cloudwatch-to-splunk lambda function. With this value false, three SSM variables are added to manage the Splunk HEC token, endpoint URI, and sourcetype.

Because SSM doesn't support empty parameters, the bogus value *** NO VALUE SET *** is pre-populated when Terraform creates the cloudwatch-to-splunk SSM. @kwessel suggests that the default for disable_subscription_filter be changed to true so that the lambda function doesn't attempt to handle logging traffic from a CloudWatch log group lacking meaningful data used to authenticate to Splunk. This creates log traffic in the lambda's own log, which doesn't bother me. However, the unsuccessful attempt to run the Splunk code produces an error that is considered a failure of the lambda function. When we start issuing alarms based on this lambda function's failure rate, this will result in false positives.

The ideal solution is for the lambda function to ignore the pre-populated bogus value and not generate an error.

My opinion is that logging is an important enough function that the infrastructure designer should have to make an explicit choice on whether he/she wants Splunk logging, alternate external logging, or no external logging other than CloudWatch itself.

JonRoma commented 3 years ago

@ddriddle, though you're no longer involved in this work, do you have an opinion one way or the other?