search

amiracle / grand_central

Grand Central logging for Cloud Services to Splunk
MIT License
36 stars 5 forks source link

Creating Cloudwatch Logs Grand Central config #19

Open grainger-ryanm opened 4 years ago

grainger-ryanm commented 4 years ago

I tried to create a Cloudwatch Logs Grand Central config for one of my accounts and it did not work. Couldn’t really tell if it was the permissions like before or something to do with the logging subscriptions to the firehose or what. Here are some screenshots with what I was trying to accomplish as well as the CFN stack that was produced.

1

2

I think it only assigned the 1 log group to the firehose? This may be a problem with firehoses, i.e. only 1 log group can be assigned to 1 firehose.

3

This is the preview of the lambda code. I’m not sure that the included zip file “splunk_cwl_firehose_processor.zip” is being read correctly, or at all. Also, the import statements highlighted here because I know AWS removed a lot of built in packages that used to be standard before (our headache came specifically from requests being removed, though it has not been taken down yet, we are working on a solution to add it back in via lambda layers).

The lambda function threw the same error each time, screenshot below.

4

Not a gzipped file: IOError Traceback (most recent call last): File "/var/task/lambda_function.py", line 191, in handler records = list(processRecords(event['records'])) File "/var/task/lambda_function.py", line 78, in processRecords data = json.loads(f.read()) File "/usr/lib64/python2.7/gzip.py", line 260, in read self._read(readsize) File "/usr/lib64/python2.7/gzip.py", line 302, in _read self._read_gzip_header() File "/usr/lib64/python2.7/gzip.py", line 196, in _read_gzip_header raise IOError, 'Not a gzipped file' IOError: Not a gzipped file

Not sure if this is running an older version of the lambda code (written in python2.7) or if the code itself won’t run because of the missing dependencies.

Things I have tried: Changing IAM permissions (changing resources to *, adding all permissions from s3, lambda, processor and delivery role all to processor role) Updating the firehoses lambda function with the newest Splunk blueprint for firehose processor (gave it the same IAM role as the one updated above) Manually subscribing the second log group to the firehose, which complained that the firehose was not active (though I confirmed that it was)

5

6

Any insight you can offer would be appreciated.

nstonesplunk commented 4 years ago

Hi there. Thanks for the detailed issue. As far as the subscription filter, it would be good to check if the deployed CloudFormation template has 2 subscription filters in the resource tab or if it was deployed with just one. The fact you are unable to add the subscription filter manually is odd. I see no attached filter to the failing group in the UI but perhaps try listing the existing filters on that group in the CLI to check if there is one stuck in a bad state? Perhaps also trying to manually add a target from the UI to the problematic log group?

Noticed another Github issue I missed mentioning a similar issue with only the first log group being affected so seems likely it's on the Splunk side.

The gzip error could have something to do with the incoming log format. Logs from CloudWatch should be delivered b64 encoded and gzipped but perhaps they are not in this case. It would be helpful to know the source and content of the log group. Seems like it just lambda logs but if there was anything special going on it could be relevant.

I will try to replicate and update next week.