search

awslabs / amazon-kinesis-scaling-utils

The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.
Apache License 2.0
338 stars 95 forks source link

Kinesis autoscaling application not working #6

Closed lexicalunit closed 9 years ago

lexicalunit commented 9 years ago

Unfortunately I don't have a more descriptive title for this ticket. I'll describe what I've done so far. I am using version 0.9.1.5.

Using Elastic Beanstalk, I installed the Kinesis Autoscaling WAR to a Web Server with Tomcat predefined configuration. Here is my autoscaling configuration:

[
    {
        "streamName":"mytest",
         "region":"us-east-1",
         "scaleOnOperation":"PUT",
         "scaleUp": {
            "scaleThresholdPct": 90,
            "scaleAfterMins": 5,
            "scaleCount": 2
        },
        "scaleDown":{
            "scaleThresholdPct": 50,
            "scaleAfterMins": 5,
            "scaleCount": 2,
            "coolOffMins": 2
        }
    },
    {
        "streamName":"mytest",
         "region":"us-east-1",
         "scaleOnOperation":"GET",
         "scaleUp": {
            "scaleThresholdPct": 90,
            "scaleAfterMins": 5,
            "scaleCount": 2
        },
        "scaleDown":{
            "scaleThresholdPct": 50,
            "scaleAfterMins": 5,
            "scaleCount": 2,
            "coolOffMins": 2
        }
    }
]

I created the mytest stream with 6 shards. There is absolutely no data being pushed through the stream. I would expect the stream to be downscaled to 1 shard over the last weekend, but this morning I'm still seeing 6 shards. I am hosting the configuration on S3, and I've set the permissions on the file such that anyone can download it. So the application should have access to grab the config.

Within the Configuration/Software Configuration settings in the EB application, I have provided values for:

The format of the config_file_url value is https://s3.amazonaws.com/my-bucket/kin-scale-config.json. The AWS keys are the keys for the account that created the Kinesis stream.

I can get the logs from the EB instance. Everything looks fine, there are no errors obvious to me. Sensitive data has been omitted:

-------------------------------------
/var/log/eb-version-deployment.log
-------------------------------------
2015-03-27 20:47:34,705 [INFO] Found the latest version manifest file <Key: elasticbeanstalk-us-east-1-OMIT,resources/environments/e-OMIT/_runtime/versions/manifest_OMIT> from bucket elasticbeanstalk-us-east-1-OMIT and prefix resources/environments/e-OMIT/_runtime/versions/
2015-03-27 20:47:34,751 [INFO] Downloaded the manifest file to /tmp/version_file_manifest
2015-03-27 20:47:35,060 [INFO] Downloaded version label First Release from s3 key <Key: elasticbeanstalk-us-east-1-OMIT,resources/environments/e-OMIT/_runtime/_versions/mytest/First Release>

-------------------------------------
/var/log/tomcat8/manager.2015-03-27.log
-------------------------------------

-------------------------------------
/var/log/httpd/error_log
-------------------------------------
[Mon Mar 30 14:01:01 2015] [notice] Digest: generating secret for digest authentication ...
[Mon Mar 30 14:01:01 2015] [notice] Digest: done
[Mon Mar 30 14:01:02 2015] [notice] Apache/2.2.29 (Unix) DAV/2 configured -- resuming normal operations

-------------------------------------
/var/log/tomcat8/localhost.2015-03-27.log
-------------------------------------

-------------------------------------
/var/log/tomcat8/catalina.2015-03-27.log
-------------------------------------

-------------------------------------
/var/log/tomcat8/tomcat8-initd.log
-------------------------------------

-------------------------------------
/var/log/tomcat8/localhost_access_log.txt
-------------------------------------

-------------------------------------
/var/log/eb-activity.log
-------------------------------------
[2015-03-27T20:53:46.781Z] INFO  [1887]  - [CMD-ConfigDeploy] : Completed activity. Result:
  Command CMD-ConfigDeploy succeeded.
[2015-03-27T20:54:41.929Z] INFO  [2071]  - [CMD-TailLogs] : Starting activity...
[2015-03-27T20:54:42.422Z] INFO  [2071]  - [CMD-TailLogs/AddonsBefore] : Starting activity...
[2015-03-27T20:54:42.422Z] INFO  [2071]  - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2015-03-27T20:54:42.423Z] INFO  [2071]  - [CMD-TailLogs/TailLogs] : Starting activity...
[2015-03-27T20:54:42.423Z] INFO  [2071]  - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity...
[2015-03-27T20:54:42.972Z] INFO  [2071]  - [CMD-TailLogs/TailLogs/TailLogs] : Completed activity.
[2015-03-27T20:54:42.973Z] INFO  [2071]  - [CMD-TailLogs/TailLogs] : Completed activity. Result:
  Command CMD-TailLogs stage 0 completed.
[2015-03-27T20:54:42.973Z] INFO  [2071]  - [CMD-TailLogs/AddonsAfter] : Starting activity...
[2015-03-27T20:54:42.973Z] INFO  [2071]  - [CMD-TailLogs/AddonsAfter] : Completed activity.
[2015-03-27T20:54:42.973Z] INFO  [2071]  - [CMD-TailLogs] : Completed activity. Result:
  Command CMD-TailLogs succeeded.
[2015-03-27T20:57:37.249Z] INFO  [2109]  - [CMD-BundleLogs] : Starting activity...
[2015-03-27T20:57:37.756Z] INFO  [2109]  - [CMD-BundleLogs/AddonsBefore] : Starting activity...
[2015-03-27T20:57:37.756Z] INFO  [2109]  - [CMD-BundleLogs/AddonsBefore] : Completed activity.
[2015-03-27T20:57:37.757Z] INFO  [2109]  - [CMD-BundleLogs/BundleLogs] : Starting activity...
[2015-03-27T20:57:37.757Z] INFO  [2109]  - [CMD-BundleLogs/BundleLogs/BundleLogs] : Starting activity...
[2015-03-27T20:57:38.148Z] INFO  [2109]  - [CMD-BundleLogs/BundleLogs/BundleLogs] : Completed activity.
[2015-03-27T20:57:38.148Z] INFO  [2109]  - [CMD-BundleLogs/BundleLogs] : Completed activity. Result:
  Command CMD-BundleLogs stage 0 completed.
[2015-03-27T20:57:38.149Z] INFO  [2109]  - [CMD-BundleLogs/AddonsAfter] : Starting activity...
[2015-03-27T20:57:38.149Z] INFO  [2109]  - [CMD-BundleLogs/AddonsAfter] : Completed activity.
[2015-03-27T20:57:38.149Z] INFO  [2109]  - [CMD-BundleLogs] : Completed activity. Result:
  Command CMD-BundleLogs succeeded.
[2015-03-27T21:04:27.206Z] INFO  [2219]  - [CMD-TailLogs] : Starting activity...
[2015-03-27T21:04:27.701Z] INFO  [2219]  - [CMD-TailLogs/AddonsBefore] : Starting activity...
[2015-03-27T21:04:27.701Z] INFO  [2219]  - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2015-03-27T21:04:27.702Z] INFO  [2219]  - [CMD-TailLogs/TailLogs] : Starting activity...
[2015-03-27T21:04:27.702Z] INFO  [2219]  - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity...
[2015-03-27T21:04:28.114Z] INFO  [2219]  - [CMD-TailLogs/TailLogs/TailLogs] : Completed activity.
[2015-03-27T21:04:28.114Z] INFO  [2219]  - [CMD-TailLogs/TailLogs] : Completed activity. Result:
  Command CMD-TailLogs stage 0 completed.
[2015-03-27T21:04:28.115Z] INFO  [2219]  - [CMD-TailLogs/AddonsAfter] : Starting activity...
[2015-03-27T21:04:28.115Z] INFO  [2219]  - [CMD-TailLogs/AddonsAfter] : Completed activity.
[2015-03-27T21:04:28.115Z] INFO  [2219]  - [CMD-TailLogs] : Completed activity. Result:
  Command CMD-TailLogs succeeded.
[2015-03-27T21:15:45.265Z] INFO  [2321]  - [CMD-ConfigDeploy] : Starting activity...
[2015-03-27T21:15:45.776Z] INFO  [2321]  - [CMD-ConfigDeploy/AddonsBefore] : Starting activity...
[2015-03-27T21:15:45.776Z] INFO  [2321]  - [CMD-ConfigDeploy/AddonsBefore] : Completed activity.
[2015-03-27T21:15:45.776Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0] : Starting activity...
[2015-03-27T21:15:45.777Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/InfraWriteConfig] : Starting activity...
[2015-03-27T21:15:45.785Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/InfraWriteConfig] : Completed activity. Result:
  Recreated directory /opt/elasticbeanstalk/deploy/configuration/.
  Generate appsource url file at /opt/elasticbeanstalk/deploy/configuration/appsourceurl.
  Generate container config file at /opt/elasticbeanstalk/deploy/configuration/containerconfiguration.
[2015-03-27T21:15:45.785Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/ConfigDeployPreHook] : Starting activity...
[2015-03-27T21:15:45.786Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/ConfigDeployPreHook/01clean.sh] : Starting activity...
[2015-03-27T21:15:45.989Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/ConfigDeployPreHook/01clean.sh] : Completed activity. Result:
  ++ /opt/elasticbeanstalk/bin/get-config container -k config_staging_dir
  + EB_CONFIG_STAGING_DIR=/tmp/deployment/config
  + rm -rf /tmp/deployment/config
  + mkdir -p /tmp/deployment/config
[2015-03-27T21:15:45.990Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/ConfigDeployPreHook/04config_generate.sh] : Starting activity...
[2015-03-27T21:15:46.651Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/ConfigDeployPreHook/04config_generate.sh] : Completed activity. Result:
  + /opt/elasticbeanstalk/containerfiles/generate_config
[2015-03-27T21:15:46.651Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0/ConfigDeployPreHook] : Completed activity. Result:
  Successfully execute directory: /opt/elasticbeanstalk/hooks/configdeploy/pre.
[2015-03-27T21:15:46.652Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage0] : Completed activity. Result:
  Command CMD-ConfigDeploy stage 0 completed.
[2015-03-27T21:15:46.652Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1] : Starting activity...
[2015-03-27T21:15:46.652Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployEnactHook] : Starting activity...
[2015-03-27T21:15:46.653Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployEnactHook/01deploy.sh] : Starting activity...
[2015-03-27T21:15:47.214Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployEnactHook/01deploy.sh] : Completed activity. Result:
  ++ /opt/elasticbeanstalk/bin/get-config container -k config_staging_dir
  + EB_CONFIG_STAGING_DIR=/tmp/deployment/config
  ++ /opt/elasticbeanstalk/bin/get-config container -k config_deploy_dir
  + EB_CONFIG_DEPLOY_DIR=/etc/sysconfig
  ++ /opt/elasticbeanstalk/bin/get-config container -k config_filename
  + EB_CONFIG_FILENAME=tomcat8
  + cp /tmp/deployment/config/tomcat8 /etc/sysconfig/tomcat8
[2015-03-27T21:15:47.214Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployEnactHook] : Completed activity. Result:
  Successfully execute directory: /opt/elasticbeanstalk/hooks/configdeploy/enact.
[2015-03-27T21:15:47.215Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployPostHook] : Starting activity...
[2015-03-27T21:15:47.215Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployPostHook/01restart.sh] : Starting activity...
[2015-03-27T21:15:49.438Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployPostHook/01restart.sh] : Completed activity. Result:
  + [[ '' != \t\r\u\e ]]
  ++ /opt/elasticbeanstalk/bin/get-config container -k tomcat_version
  + TOMCAT_VERSION=8
  + TOMCAT_NAME=tomcat8
  + /usr/bin/monit unmonitor tomcat
  + /etc/init.d/tomcat8 stop
  Stopping tomcat8: [  OK  ]
+ /etc/init.d/tomcat8 start
  Starting tomcat8: [  OK  ]
+ /usr/bin/monit monitor tomcat
[2015-03-27T21:15:49.438Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1/ConfigDeployPostHook] : Completed activity. Result:
  Successfully execute directory: /opt/elasticbeanstalk/hooks/configdeploy/post.
[2015-03-27T21:15:49.439Z] INFO  [2321]  - [CMD-ConfigDeploy/ConfigDeployStage1] : Completed activity. Result:
  Command CMD-ConfigDeploy stage 1 completed.
[2015-03-27T21:15:49.439Z] INFO  [2321]  - [CMD-ConfigDeploy/AddonsAfter] : Starting activity...
[2015-03-27T21:15:49.439Z] INFO  [2321]  - [CMD-ConfigDeploy/AddonsAfter] : Completed activity.
[2015-03-27T21:15:49.439Z] INFO  [2321]  - [CMD-ConfigDeploy] : Completed activity. Result:
  Command CMD-ConfigDeploy succeeded.
[2015-03-30T14:17:00.023Z] INFO  [19451] - [CMD-TailLogs] : Starting activity...
[2015-03-30T14:17:00.530Z] INFO  [19451] - [CMD-TailLogs/AddonsBefore] : Starting activity...
[2015-03-30T14:17:00.530Z] INFO  [19451] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2015-03-30T14:17:00.530Z] INFO  [19451] - [CMD-TailLogs/TailLogs] : Starting activity...
[2015-03-30T14:17:00.530Z] INFO  [19451] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity...

-------------------------------------
/var/log/httpd/elasticbeanstalk-access_log
-------------------------------------

-------------------------------------
/var/log/httpd/access_log
-------------------------------------

-------------------------------------
/var/log/eb-commandprocessor.log
-------------------------------------
[2015-03-27T20:57:37.756Z] DEBUG [2109]  : Loaded definition of Command CMD-BundleLogs.
[2015-03-27T20:57:37.756Z] INFO  [2109]  : Executing command CMD-BundleLogs activities...
[2015-03-27T20:57:37.756Z] DEBUG [2109]  : Setting environment variables..
[2015-03-27T20:57:37.756Z] INFO  [2109]  : Running AddonsBefore for command CMD-BundleLogs...
[2015-03-27T20:57:37.757Z] DEBUG [2109]  : Running stages of Command CMD-BundleLogs from stage 0 to stage 0...
[2015-03-27T20:57:37.757Z] INFO  [2109]  : Running stage 0 of command CMD-BundleLogs...
[2015-03-27T20:57:37.757Z] DEBUG [2109]  : Loaded 1 actions for stage 0.
[2015-03-27T20:57:37.757Z] INFO  [2109]  : Running 1 of 1 actions: BundleLogs...
[2015-03-27T20:57:38.148Z] INFO  [2109]  : Running AddonsAfter for command CMD-BundleLogs...
[2015-03-27T20:57:38.149Z] INFO  [2109]  : Command CMD-BundleLogs succeeded!
[2015-03-27T20:57:38.150Z] INFO  [2109]  : Command processor returning results:
{"status":"SUCCESS","api_version":"1.0","truncated":"false","results":[{"status":"SUCCESS","msg":"","returncode":0,"events":[{"msg":"[Instance: i-OMIT] Successfully finished bundling 26 log(s)","severity":"INFO","timestamp":1427489858150}]}]}
[2015-03-27T21:04:27.200Z] DEBUG [2219]  : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2015-03-27T21:04:27.201Z] INFO  [2219]  : Received command CMD-TailLogs: {"command_name":"CMD-TailLogs","execution_data":"{\"aws_access_key_id\":\"OMIT\",\"policy\":\"OMIT==\",\"security_token\":\"OMIT\",\"signature\":\"OMIT\"}","request_id":"OMIT","resource_name":"AWSEBAutoScalingGroup","data":"OMIT","command_timeout":"600","api_version":"1.0"}
[2015-03-27T21:04:27.201Z] DEBUG [2219]  : Checking if the command processor should execute.
[2015-03-27T21:04:27.205Z] DEBUG [2219]  : Checking whether the command is applicable to instance (i-OMIT)..
[2015-03-27T21:04:27.205Z] INFO  [2219]  : Command is applicable to this instance (i-OMIT)..
[2015-03-27T21:04:27.205Z] DEBUG [2219]  : Checking if the received command stage is valid..
[2015-03-27T21:04:27.205Z] INFO  [2219]  : No stage_num in command. Valid stage..
[2015-03-27T21:04:27.205Z] INFO  [2219]  : Command processor should execute command.
[2015-03-27T21:04:27.205Z] DEBUG [2219]  : Storing current stage..
[2015-03-27T21:04:27.205Z] DEBUG [2219]  : Stage_num does not exist. Not saving null stage. Returning..
[2015-03-27T21:04:27.206Z] INFO  [2219]  : Executing command: CMD-TailLogs...
[2015-03-27T21:04:27.206Z] DEBUG [2219]  : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2015-03-27T21:04:27.206Z] DEBUG [2219]  : Refreshing metadata..
[2015-03-27T21:04:27.698Z] DEBUG [2219]  : Refreshed environment metadata.
[2015-03-27T21:04:27.698Z] DEBUG [2219]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_ContainerConfigFileContent||commands..
[2015-03-27T21:04:27.700Z] DEBUG [2219]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_API||_Commands..
[2015-03-27T21:04:27.701Z] INFO  [2219]  : Found enabled addons: [].
[2015-03-27T21:04:27.701Z] DEBUG [2219]  : Loaded definition of Command CMD-TailLogs.
[2015-03-27T21:04:27.701Z] INFO  [2219]  : Executing command CMD-TailLogs activities...
[2015-03-27T21:04:27.701Z] DEBUG [2219]  : Setting environment variables..
[2015-03-27T21:04:27.701Z] INFO  [2219]  : Running AddonsBefore for command CMD-TailLogs...
[2015-03-27T21:04:27.702Z] DEBUG [2219]  : Running stages of Command CMD-TailLogs from stage 0 to stage 0...
[2015-03-27T21:04:27.702Z] INFO  [2219]  : Running stage 0 of command CMD-TailLogs...
[2015-03-27T21:04:27.702Z] DEBUG [2219]  : Loaded 1 actions for stage 0.
[2015-03-27T21:04:27.702Z] INFO  [2219]  : Running 1 of 1 actions: TailLogs...
[2015-03-27T21:04:28.114Z] INFO  [2219]  : Running AddonsAfter for command CMD-TailLogs...
[2015-03-27T21:04:28.115Z] INFO  [2219]  : Command CMD-TailLogs succeeded!
[2015-03-27T21:04:28.116Z] INFO  [2219]  : Command processor returning results:
{"status":"SUCCESS","api_version":"1.0","truncated":"false","results":[{"status":"SUCCESS","msg":"","returncode":0,"events":[{"msg":"[Instance: i-OMIT] Successfully finished tailing 14 log(s)","severity":"INFO","timestamp":1427490268116}]}]}
[2015-03-27T21:15:45.231Z] DEBUG [2321]  : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2015-03-27T21:15:45.231Z] INFO  [2321]  : Received command CMD-ConfigDeploy: {"command_name":"CMD-ConfigDeploy","request_id":"OMIT","resource_name":"AWSEBAutoScalingGroup","command_timeout":"600","api_version":"1.0"}
[2015-03-27T21:15:45.231Z] DEBUG [2321]  : Checking if the command processor should execute.
[2015-03-27T21:15:45.264Z] DEBUG [2321]  : Checking whether the command is applicable to instance (i-OMIT)..
[2015-03-27T21:15:45.264Z] INFO  [2321]  : Command is applicable to this instance (i-OMIT)..
[2015-03-27T21:15:45.264Z] DEBUG [2321]  : Checking if the received command stage is valid..
[2015-03-27T21:15:45.264Z] INFO  [2321]  : No stage_num in command. Valid stage..
[2015-03-27T21:15:45.264Z] INFO  [2321]  : Command processor should execute command.
[2015-03-27T21:15:45.264Z] DEBUG [2321]  : Storing current stage..
[2015-03-27T21:15:45.264Z] DEBUG [2321]  : Stage_num does not exist. Not saving null stage. Returning..
[2015-03-27T21:15:45.265Z] INFO  [2321]  : Executing command: CMD-ConfigDeploy...
[2015-03-27T21:15:45.266Z] DEBUG [2321]  : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2015-03-27T21:15:45.266Z] DEBUG [2321]  : Refreshing metadata..
[2015-03-27T21:15:45.772Z] DEBUG [2321]  : Refreshed environment metadata.
[2015-03-27T21:15:45.773Z] DEBUG [2321]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_ContainerConfigFileContent||commands..
[2015-03-27T21:15:45.774Z] DEBUG [2321]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_API||_Commands..
[2015-03-27T21:15:45.775Z] INFO  [2321]  : Found enabled addons: [].
[2015-03-27T21:15:45.776Z] DEBUG [2321]  : Loaded definition of Command CMD-ConfigDeploy.
[2015-03-27T21:15:45.776Z] INFO  [2321]  : Executing command CMD-ConfigDeploy activities...
[2015-03-27T21:15:45.776Z] DEBUG [2321]  : Setting environment variables..
[2015-03-27T21:15:45.776Z] INFO  [2321]  : Running AddonsBefore for command CMD-ConfigDeploy...
[2015-03-27T21:15:45.776Z] DEBUG [2321]  : Running stages of Command CMD-ConfigDeploy from stage 0 to stage 1...
[2015-03-27T21:15:45.776Z] INFO  [2321]  : Running stage 0 of command CMD-ConfigDeploy...
[2015-03-27T21:15:45.776Z] DEBUG [2321]  : Loaded 2 actions for stage 0.
[2015-03-27T21:15:45.777Z] INFO  [2321]  : Running 1 of 2 actions: InfraWriteConfig...
[2015-03-27T21:15:45.785Z] INFO  [2321]  : Running 2 of 2 actions: ConfigDeployPreHook...
[2015-03-27T21:15:46.652Z] INFO  [2321]  : Running stage 1 of command CMD-ConfigDeploy...
[2015-03-27T21:15:46.652Z] DEBUG [2321]  : Loaded 2 actions for stage 1.
[2015-03-27T21:15:46.652Z] INFO  [2321]  : Running 1 of 2 actions: ConfigDeployEnactHook...
[2015-03-27T21:15:47.215Z] INFO  [2321]  : Running 2 of 2 actions: ConfigDeployPostHook...
[2015-03-27T21:15:49.439Z] INFO  [2321]  : Running AddonsAfter for command CMD-ConfigDeploy...
[2015-03-27T21:15:49.439Z] INFO  [2321]  : Command CMD-ConfigDeploy succeeded!
[2015-03-27T21:15:49.440Z] INFO  [2321]  : Command processor returning results:
{"status":"SUCCESS","api_version":"1.0","truncated":"false","results":[{"status":"SUCCESS","msg":"","returncode":0,"events":[]}]}
[2015-03-30T14:17:00.018Z] DEBUG [19451] : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2015-03-30T14:17:00.018Z] INFO  [19451] : Received command CMD-TailLogs: {"command_name":"CMD-TailLogs","execution_data":"{\"aws_access_key_id\":\"OMIT\",\"policy\":\"OMIT\",\"security_token\":\"OMIT\",\"signature\":\"OMIT\"}","request_id":"OMIT","resource_name":"AWSEBAutoScalingGroup","data":"OMIT","command_timeout":"600","api_version":"1.0"}
[2015-03-30T14:17:00.018Z] DEBUG [19451] : Checking if the command processor should execute.
[2015-03-30T14:17:00.022Z] DEBUG [19451] : Checking whether the command is applicable to instance (i-OMIT)..
[2015-03-30T14:17:00.022Z] INFO  [19451] : Command is applicable to this instance (i-OMIT)..
[2015-03-30T14:17:00.022Z] DEBUG [19451] : Checking if the received command stage is valid..
[2015-03-30T14:17:00.022Z] INFO  [19451] : No stage_num in command. Valid stage..
[2015-03-30T14:17:00.022Z] INFO  [19451] : Command processor should execute command.
[2015-03-30T14:17:00.022Z] DEBUG [19451] : Storing current stage..
[2015-03-30T14:17:00.022Z] DEBUG [19451] : Stage_num does not exist. Not saving null stage. Returning..
[2015-03-30T14:17:00.023Z] INFO  [19451] : Executing command: CMD-TailLogs...
[2015-03-30T14:17:00.023Z] DEBUG [19451] : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2015-03-30T14:17:00.023Z] DEBUG [19451] : Refreshing metadata..
[2015-03-30T14:17:00.526Z] DEBUG [19451] : Refreshed environment metadata.
[2015-03-30T14:17:00.527Z] DEBUG [19451] : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_ContainerConfigFileContent||commands..
[2015-03-30T14:17:00.528Z] DEBUG [19451] : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_API||_Commands..
[2015-03-30T14:17:00.529Z] INFO  [19451] : Found enabled addons: [].
[2015-03-30T14:17:00.529Z] DEBUG [19451] : Loaded definition of Command CMD-TailLogs.
[2015-03-30T14:17:00.529Z] INFO  [19451] : Executing command CMD-TailLogs activities...
[2015-03-30T14:17:00.529Z] DEBUG [19451] : Setting environment variables..
[2015-03-30T14:17:00.529Z] INFO  [19451] : Running AddonsBefore for command CMD-TailLogs...
[2015-03-30T14:17:00.530Z] DEBUG [19451] : Running stages of Command CMD-TailLogs from stage 0 to stage 0...
[2015-03-30T14:17:00.530Z] INFO  [19451] : Running stage 0 of command CMD-TailLogs...
[2015-03-30T14:17:00.530Z] DEBUG [19451] : Loaded 1 actions for stage 0.
[2015-03-30T14:17:00.530Z] INFO  [19451] : Running 1 of 1 actions: TailLogs...

-------------------------------------
/var/log/httpd/elasticbeanstalk-error_log
-------------------------------------

-------------------------------------
/var/log/tomcat8/host-manager.2015-03-27.log
-------------------------------------

-------------------------------------
/var/log/tomcat8/catalina.out
-------------------------------------
IanMeyers commented 9 years ago

Hello,

Thanks for the complete log. You don't need to put in configuration values for access & secret key, and in fact you shouldn't as this isn't very secure. Instead, run your Elastic Beanstalk application with an EC2 Role that will give it the required permissions, such as the ability to read cloudwatch, kinesis, dynamo db etc. Then you just need to configure parameter 'config-file-url', and use a format such as 's3://mybucket/my-config-file.json'. Lastly, ensure that the config file URL is readable by this EC2 Instance Role.

Hopefully that will get you up and running!

Thx,

Ian

lexicalunit commented 9 years ago

Oops, I incorrectly used underscores in the key name config-file-url. I'll fix that.

As for required permissions. Do you mean IAM Roles or EC2 Security Groups? I'm not sure what a "EC2 Role" is.

IanMeyers commented 9 years ago

Yes, sorry for the confusion - I meant the IAM Role that is applied to the EC2 instance in the Elastic Beanstalk Fleet.

lexicalunit commented 9 years ago

Sweet! It's working :)

Initially I created a very open policy to avoid permissions issues. I am now removing Action items to restrict the policy to only what is necessary. Right now I have something like

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "...",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "..."
    },
    {
      "Sid": "...",
      "Action": [
        "cloudwatch:GetMetricStatistics"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "...",
      "Action": [
        "kinesis:GetShardIterator",
        "kinesis:MergeShards",
        "kinesis:SplitShard"
      ],
      "Effect": "Allow",
      "Resource": "..."
    }
  ]
}

I'll eventually discover through experimentation, but would much appreciate it if you could let me know if there is anything in this policy that I'm missing. Might also be something good to add to the README or wiki perhaps?

IanMeyers commented 9 years ago

Glad to hear it. I agree that we need to show a policy that is the minimum for this module to run, and I'm going to leave this issue open until I've had time to put this together. In the meantime you can start with a policy that is read only on CloudWatch, S3 and read only plus merge/split shards in Kinesis.

lexicalunit commented 9 years ago

:+1: Y'all rock! Having this functionality is going to be awesome for us.

lexicalunit commented 9 years ago

Just to note, we had to add

        "kinesis:DescribeStream",

To the above policy.

avram commented 9 years ago

I went ahead and wrote up some more detail on the permission requirements in a fork of the repo: https://github.com/scopely/amazon-kinesis-scaling-utils#iam-role