theburningmonk / lambda-logging-demo

Demo for shipping logs to ELK stack, and to auto-subscribe new log groups
91 stars 31 forks source link

Log subscribe not working #3

Open Humen opened 6 years ago

Humen commented 6 years ago

Hi, I cloned this project and deployed and I noticed that the subscribe function has no triggers. I would expect a cloudwatch event to be shown as a trigger.

I then tested if it actually works by creating a hello-world function via serverless and the log group does not have the subscription

dpapukchiev commented 6 years ago

Did you enable CloudTrail? If you haven't you just need to add a trail and give it some time. After it's ready, you can test the whole flow. Subscribing the log group is dependent on the speed of the CloudTrail events, in my experience it took less than a minute but some times more. Eventually the log groups are subscribed to the lambda function. What I can't get to work is subscribing all prior logs. I am getting a

InvalidParameterException: Could not execute the lambda function. Make sure you have given
CloudWatch Logs permission to execute your function.
hitenkaram commented 5 years ago

@dpapukchiev I'm also facing the same issue, did you manage to get any solution for the above permission problem?

renuez commented 5 years ago

@dpapukchiev double check the configuration in process_all.js. The funcName is not related to any bigmouth Lambdas, it's related to the actual logzio log shipping Lambda, see my working example below :

const region = "eu-central-1";
const accountId = "12345678910";
const funcName = "cloudwatchlogs-to-logzio-dev-ship-logs-to-logzio";
const retentionDays = 7;       // change this if you want
const prefix = '/aws/lambda';  // use '/' if you want to process every log group

You find the Lambda name either in AWS Management console or your sneak the structure in your current serverless.yml of the dest_func env variable:

functions:
  subscribe:
    handler: functions/subscribe/handler.handler
    description: Subscribe logs to the ship-logs-to-logzio function
    memorySize: 128
    environment:
      dest_func: "${self:service}-${self:provider.stage}-ship-logs-to-logzio"
      account_id: "#{AWS::AccountId}"
      prefix: "/aws/lambda"