search

SumoLogic / sumologic-aws-lambda

A collection of lambda functions to collect data from Cloudwatch, Kinesis, VPC Flow logs, S3, security-hub and AWS Inspector
Other
152 stars 134 forks source link

Lambda connector - create subscription fails due to throttling #109

Open craigwatkinson opened 4 years ago

craigwatkinson commented 4 years ago

When there are a large number of existing log groups, and USE_EXISTING_LOG_GROUPS=True then the putSubscriptionFilter AWS API call sometimes fails due to hitting throttling limits. We see errors similar to the one below:

2020-01-21T12:23:02.436Z    a30ab1f3-8cfe-4f5c-84e4-0d404da581cc    INFO    Error in subscribing /aws/lambda/dev-somefunction { LimitExceededException: Resource limit exceeded.
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
  message: 'Resource limit exceeded.',
  code: 'LimitExceededException',
  time: 2020-01-21T12:23:02.377Z,
  requestId: 'a2066014-38a4-43b3-934e-0309185b910f',
  statusCode: 400,
  retryable: false,
  retryDelay: 38.911558105243586 }

Re-trying the invocation usually causes the throttle limits to be hit in the same places, so doesn't help to resolve the situation.

himanshu219 commented 4 years ago

Can you try with more granular regex like in your case \/aws\/lambda\/dev.* Unfortunately we do not check whether the log group is already subscribed currently and AWS APIs do not provide a way to filter that in a single call so it will fail when one retries.