search

amazon-archives / cloudwatch-logs-subscription-consumer

A specialized Amazon Kinesis stream reader (based on the Amazon Kinesis Connector Library) that can help you deliver data from Amazon CloudWatch Logs to any other system in near real-time using a CloudWatch Logs Subscription Filter.
Other
397 stars 152 forks source link

Cloudformation template failure #1

Closed tintheparkin closed 8 years ago

tintheparkin commented 8 years ago

Hi,

My colleague and I have been trying to begin a testing environment to see if this will be useful enough for our daily working. We have come across an error when initially deploying the cloudfromation template supplied /configuration/cloudformation/cwl-elasticsearch.template. Is this something you have come across before? We've logged into the created instance and cfn-init is indeed accessible on the instance. below details the error;

16:27:58 UTC+0100 CREATE_FAILED AWS::CloudFormation::WaitCondition WaitCondition WaitCondition received failed message: 'failed to run cfn-init' for uniqueId: i-c8939d0a Physical ID:arn:aws:cloudformation:us-west-1:532578262755:stack/test-stack-cf/ade64480-3abc-11e5-82fb-500c335a70e0/WaitHandle

tintheparkin commented 8 years ago

I've checked the cfn-init.log and there seems to be an issue with locating the log group?

Apologies for the size of message!

2015-08-04 15:50:09,525 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-west-1.amazonaws.com
2015-08-04 15:50:09,526 [DEBUG] Describing resource ElasticsearchServer in stack arn:aws:cloudformation:us-west-1:532578262755:stack/teststack----/d437f630-3abf-11e5-81c3-508813d72498
2015-08-04 15:50:09,849 [INFO] -----------------------Starting build-----------------------
2015-08-04 15:50:10,139 [DEBUG] Not setting a reboot trigger as scheduling support is not available
2015-08-04 15:50:10,143 [INFO] Running configSets: default
2015-08-04 15:50:10,143 [INFO] Running configSet default
2015-08-04 15:50:10,144 [INFO] Running config config
2015-08-04 15:50:10,145 [DEBUG] Querying for version of package https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.6.0.noarch.rpm
2015-08-04 15:50:13,415 [DEBUG] RPM stdout: elasticsearch|1.6.0-1
2015-08-04 15:50:13,415 [DEBUG] RPM stderr:
2015-08-04 15:50:13,422 [DEBUG] Installing [u'https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.6.0.noarch.rpm'] via RPM
2015-08-04 15:50:17,115 [DEBUG] RPM output: Creating elasticsearch group... OK
Creating elasticsearch user... OK
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
 sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
 sudo service elasticsearch start

2015-08-04 15:50:18,985 [DEBUG] Installing/updating [u'nginx', u'awslogs', u'httpd'] via yum
2015-08-04 15:50:24,651 [INFO] Yum installed [u'nginx', u'awslogs', u'httpd']
2015-08-04 15:50:24,651 [DEBUG] No groups specified
2015-08-04 15:50:24,652 [DEBUG] No users specified
2015-08-04 15:50:24,652 [DEBUG] No sources specified
2015-08-04 15:50:24,652 [DEBUG] /etc/awslogs/awscli.conf already exists
2015-08-04 15:50:24,652 [DEBUG] Moving /etc/awslogs/awscli.conf to /etc/awslogs/awscli.conf.bak
2015-08-04 15:50:24,652 [DEBUG] Writing content to /etc/awslogs/awscli.conf
2015-08-04 15:50:24,652 [DEBUG] No mode specified for /etc/awslogs/awscli.conf
2015-08-04 15:50:24,653 [DEBUG] /etc/awslogs/awslogs.conf already exists
2015-08-04 15:50:24,653 [DEBUG] Moving /etc/awslogs/awslogs.conf to /etc/awslogs/awslogs.conf.bak
2015-08-04 15:50:24,653 [DEBUG] Writing content to /etc/awslogs/awslogs.conf
2015-08-04 15:50:24,653 [DEBUG] Setting mode for /etc/awslogs/awslogs.conf to 000444
2015-08-04 15:50:24,653 [DEBUG] Setting owner 0 and group 0 for /etc/awslogs/awslogs.conf
2015-08-04 15:50:24,653 [DEBUG] /etc/elasticsearch/elasticsearch.yml already exists
2015-08-04 15:50:24,654 [DEBUG] Moving /etc/elasticsearch/elasticsearch.yml to /etc/elasticsearch/elasticsearch.yml.bak
2015-08-04 15:50:24,654 [DEBUG] Writing content to /etc/elasticsearch/elasticsearch.yml
2015-08-04 15:50:24,654 [DEBUG] No mode specified for /etc/elasticsearch/elasticsearch.yml
2015-08-04 15:50:24,654 [DEBUG] Writing content to /etc/logrotate.d/cloudwatch-logs-subscription-consumer
2015-08-04 15:50:24,654 [DEBUG] No mode specified for /etc/logrotate.d/cloudwatch-logs-subscription-consumer
2015-08-04 15:50:24,654 [DEBUG] Writing content to /etc/logrotate.d/elasticsearch
2015-08-04 15:50:24,654 [DEBUG] No mode specified for /etc/logrotate.d/elasticsearch
2015-08-04 15:50:24,655 [DEBUG] /etc/logrotate.d/nginx already exists
2015-08-04 15:50:24,655 [DEBUG] Moving /etc/logrotate.d/nginx to /etc/logrotate.d/nginx.bak
2015-08-04 15:50:24,655 [DEBUG] Writing content to /etc/logrotate.d/nginx
2015-08-04 15:50:24,655 [DEBUG] No mode specified for /etc/logrotate.d/nginx
2015-08-04 15:50:24,655 [DEBUG] Parent directory /pipstuff does not exist, creating
2015-08-04 15:50:24,655 [DEBUG] Writing content to /pipstuff/get-pip.py
2015-08-04 15:50:24,655 [DEBUG] Retrieving contents from https://bootstrap.pypa.io/get-pip.py
2015-08-04 15:50:24,885 [DEBUG] No mode specified for /pipstuff/get-pip.py
2015-08-04 15:50:24,886 [DEBUG] Running command 00_copyNginxConfig
2015-08-04 15:50:24,886 [DEBUG] No test for command 00_copyNginxConfig
2015-08-04 15:50:24,898 [INFO] Command 00_copyNginxConfig succeeded
2015-08-04 15:50:24,898 [DEBUG] Command 00_copyNginxConfig output:
2015-08-04 15:50:24,898 [DEBUG] Running command 01_copyElasticsearchTemplates
2015-08-04 15:50:24,898 [DEBUG] No test for command 01_copyElasticsearchTemplates
2015-08-04 15:50:24,901 [INFO] Command 01_copyElasticsearchTemplates succeeded
2015-08-04 15:50:24,901 [DEBUG] Command 01_copyElasticsearchTemplates output:
2015-08-04 15:50:24,901 [DEBUG] Running command 02_updateESConfigForNode
2015-08-04 15:50:24,902 [DEBUG] No test for command 02_updateESConfigForNode
2015-08-04 15:50:24,922 [INFO] Command 02_updateESConfigForNode succeeded
2015-08-04 15:50:24,922 [DEBUG] Command 02_updateESConfigForNode output:
2015-08-04 15:50:24,922 [DEBUG] Running command 03_installESPlugins
2015-08-04 15:50:24,922 [DEBUG] No test for command 03_installESPlugins
2015-08-04 15:50:31,686 [INFO] Command 03_installESPlugins succeeded
2015-08-04 15:50:31,686 [DEBUG] Command 03_installESPlugins output: -> Installing elasticsearch/elasticsearch-cloud-aws/2.6.0...
Trying http://download.elasticsearch.org/elasticsearch/elasticsearch-cloud-aws/elasticsearch-cloud-aws-2.6.0.zip...
Downloading .......................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed elasticsearch/elasticsearch-cloud-aws/2.6.0 into /usr/share/elasticsearch/plugins/cloud-aws
-> Installing lmenezes/elasticsearch-kopf/1.5.5...
Trying http://download.elasticsearch.org/lmenezes/elasticsearch-kopf/elasticsearch-kopf-1.5.5.zip...
Trying http://search.maven.org/remotecontent?filepath=lmenezes/elasticsearch-kopf/1.5.5/elasticsearch-kopf-1.5.5.zip...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/lmenezes/elasticsearch-kopf/1.5.5/elasticsearch-kopf-1.5.5.zip...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/1.5.5.zip...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip...
Downloading .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed lmenezes/elasticsearch-kopf/1.5.5 into /usr/share/elasticsearch/plugins/kopf
Identified as a _site plugin, moving to _site structure ...

2015-08-04 15:50:31,686 [DEBUG] Running command 04_deleteSubscriptionFilter
2015-08-04 15:50:31,687 [DEBUG] No test for command 04_deleteSubscriptionFilter
2015-08-04 15:50:32,714 [ERROR] Command 04_deleteSubscriptionFilter (aws logs delete-subscription-filter --log-group-name "group" --region "us-west-1" --filter-name $(aws logs describe-subscription-filters --log-group-name group --region us-west-1 --filter-name-prefix "cwl-cfn-es-" | grep filterName | awk -F \" '{ print $4 };' )) failed
2015-08-04 15:50:32,715 [DEBUG] Command 04_deleteSubscriptionFilter output:
A client error (ResourceNotFoundException) occurred when calling the DescribeSubscriptionFilters operation: The specified log group does not exist.
usage: aws [options] <command> <subcommand> [parameters]
aws: error: argument --filter-name: expected one argument

2015-08-04 15:50:32,715 [INFO] ignoreErrors set to true, continuing build
2015-08-04 15:50:32,715 [DEBUG] Running command 05_putSubscriptionFilter
2015-08-04 15:50:32,715 [DEBUG] No test for command 05_putSubscriptionFilter
2015-08-04 15:50:33,281 [ERROR] Command 05_putSubscriptionFilter (declare -A FILTERS=([Amazon VPC Flow Logs]="[version, account_id, interface_id, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, action, log_status]" [AWS Lambda]="[timestamp=*Z, request_id=\"*-*\", event]" [AWS CloudTrail]="" [Custom]="" ) && aws logs put-subscription-filter --log-group-name "group" --filter-name "cwl-cfn-es-teststack---KinesisSubscriptionStream-13GZ6EN283GSM" --filter-pattern "${FILTERS["Custom"]}" --region "us-west-1" --destination-arn "arn:aws:kinesis:us-west-1:532578262755:stream/teststack---KinesisSubscriptionStream-13GZ6EN283GSM" --role-arn "arn:aws:iam::532578262755:role/teststack---CloudWatchLogsKinesisRole-17YNUWE56Q79R") failed
2015-08-04 15:50:33,281 [DEBUG] Command 05_putSubscriptionFilter output:
A client error (ResourceNotFoundException) occurred when calling the PutSubscriptionFilter operation: The specified log group does not exist.

2015-08-04 15:50:33,282 [ERROR] Error encountered during build of config: Command 05_putSubscriptionFilter failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 517, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 248, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 113, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 05_putSubscriptionFilter failed
2015-08-04 15:50:33,282 [ERROR] -----------------------BUILD FAILED!------------------------
2015-08-04 15:50:33,285 [ERROR] Unhandled exception during build: Command 05_putSubscriptionFilter failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 123, in <module>
    worklog.build(detail.metadata, configSets)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 118, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 505, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 517, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 248, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 113, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 05_putSubscriptionFilter failed
[ec2-user@ip-172-31-31-22 ~]$
dvassallo commented 8 years ago

Hello,

It seems that you specified a log group called "group" in your CloudFormation template parameters. Is this an existing CloudWatch Logs log group?

Note that this CloudFormation stack is intended to consume log data from an already existing CloudWatch Logs log group and have that data indexed on an Elasticsearch cluster running on EC2. You can have log data in a CloudWatch Logs log group either by installing the CloudWatch Logs agent to submit application logs from your own hosts, or by having logs from certain AWS services delivered to your CloudWatch Logs account. Example: VPC Flow logs and CloudTrail logs.

Also note that your CloudFormation stack should be started in the same AWS region where the CloudWatch Logs log group exists.

You can check which log groups are present in your AWS account by visiting this link after you've signed in your AWS account: https://console.aws.amazon.com/cloudwatch/home?#logs: