search

aws-samples / aws-lambda-lifecycle-hooks-function

Using Auto Scaling lifecycle hooks, Lambda, and EC2 Run Command
Apache License 2.0
102 stars 60 forks source link

Index Out of Rang errir #1

Open nathaniela opened 7 years ago

nathaniela commented 7 years ago

Though the backup procedure works, there is an error in the cloudwatch logs of the lambda function:

[ERROR] 2017-07-12T07:22:50.632Z e8984d42-66d2-11e7-a7af-510b0421530a Error: list index out of range

Here is the full log:


[DEBUG] 2017-07-12T07:22:49.907Z    Loading JSON file: /var/runtime/botocore/data/endpoints.json
[DEBUG] 2017-07-12T07:22:50.31Z Loading JSON file: /var/runtime/botocore/data/ssm/2014-11-06/service-2.json
[DEBUG] 2017-07-12T07:22:50.64Z Loading JSON file: /var/runtime/botocore/data/_retry.json
[DEBUG] 2017-07-12T07:22:50.65Z Registering retry handlers for service: ssm
[DEBUG] 2017-07-12T07:22:50.68Z Event creating-client-class.ssm: calling handler <function add_generate_presigned_url at 0x7faf2e12e848>
[DEBUG] 2017-07-12T07:22:50.69Z The s3 config key is not a dictionary type, ignoring its value of: None
[DEBUG] 2017-07-12T07:22:50.70Z Setting ssm timeout as (60, 60)
START RequestId: e8984d42-66d2-11e7-a7af-510b0421530a Version: $LATEST
[INFO]  2017-07-12T07:22:50.71Z e8984d42-66d2-11e7-a7af-510b0421530a
{
    "account": "350785128185",
    "region": "us-east-1",
    "detail": {
        "LifecycleHookName": "ASGBackup",
        "AutoScalingGroupName": "awseb-e-mam8pnazxz-stack-AWSEBAutoScalingGroup-1RRZHOXULUX78",
        "LifecycleActionToken": "a860bb0a-3dfb-4cfc-b64f-86d228aea689",
        "LifecycleTransition": "autoscaling:EC2_INSTANCE_TERMINATING",
        "EC2InstanceId": "i-057beff453ba8887e"
    },
    "detail-type": "EC2 Instance-terminate Lifecycle Action",
    "source": "aws.autoscaling",
    "version": "0",
    "time": "2017-07-12T07:22:48Z",
    "id": "3e0b0358-9730-44c5-89a2-096e441c0c23",
    "resources": [
        "arn:aws:autoscaling:us-east-1:350785128185:autoScalingGroup:a8d17511-eec5-4b17-866b-ea0d5f1b0c36:autoScalingGroupName/awseb-e-mam8pnazxz-stack-AWSEBAutoScalingGroup-1RRZHOXULUX78"
    ]
}

[DEBUG] 2017-07-12T07:22:50.71Z e8984d42-66d2-11e7-a7af-510b0421530a    Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG] 2017-07-12T07:22:50.72Z e8984d42-66d2-11e7-a7af-510b0421530a    Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.ListDocuments'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG] 2017-07-12T07:22:50.72Z e8984d42-66d2-11e7-a7af-510b0421530a    Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG] 2017-07-12T07:22:50.72Z e8984d42-66d2-11e7-a7af-510b0421530a    Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG] 2017-07-12T07:22:50.73Z e8984d42-66d2-11e7-a7af-510b0421530a    Calculating signature using v4 auth.
[DEBUG] 2017-07-12T07:22:50.76Z e8984d42-66d2-11e7-a7af-510b0421530a    CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.ListDocuments

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
769c5616cb57fca6e74861906824d48410e9a1c6052187a8c2afaeda27fb55b6
[DEBUG] 2017-07-12T07:22:50.76Z e8984d42-66d2-11e7-a7af-510b0421530a    StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
78f369959524afc6a18bbaba60b947d6c02a25f147319e2cbbf673cb7ab4b987
[DEBUG] 2017-07-12T07:22:50.77Z e8984d42-66d2-11e7-a7af-510b0421530a    Signature:
dabe509b02c447b3d3612fc0837ba6ccc14c0fafa6e4be6518328e97a3b83840
[DEBUG] 2017-07-12T07:22:50.100Z    e8984d42-66d2-11e7-a7af-510b0421530a    Sending http request: <PreparedRequest [POST]>
[INFO]  2017-07-12T07:22:50.100Z    e8984d42-66d2-11e7-a7af-510b0421530a    Starting new HTTPS connection (1): ssm.us-east-1.amazonaws.com
[DEBUG] 2017-07-12T07:22:50.303Z    e8984d42-66d2-11e7-a7af-510b0421530a    "POST / HTTP/1.1" 200 167
[DEBUG] 2017-07-12T07:22:50.304Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response headers: {'x-amzn-requestid': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG] 2017-07-12T07:22:50.316Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response body:
{
    "DocumentIdentifiers": [
        {
            "DocumentType": "Command",
            "DocumentVersion": "1",
            "Name": "ASGLogBackup",
            "Owner": "350785128185",
            "PlatformTypes": [
                "Linux"
            ],
            "SchemaVersion": "1.2"
        }
    ]
}

[DEBUG] 2017-07-12T07:22:50.317Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG] 2017-07-12T07:22:50.317Z    e8984d42-66d2-11e7-a7af-510b0421530a    No retry needed.
[INFO]  2017-07-12T07:22:50.317Z    e8984d42-66d2-11e7-a7af-510b0421530a    Documents list: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'350785128185', u'SchemaVersion': u'1.2'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'HTTPHeaders': {'x-amzn-requestid': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}}}
[INFO]  2017-07-12T07:22:50.317Z    e8984d42-66d2-11e7-a7af-510b0421530a    Documents exists: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'350785128185', u'SchemaVersion': u'1.2'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'HTTPHeaders': {'x-amzn-requestid': 'e961de9e-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}}}
[DEBUG] 2017-07-12T07:22:50.318Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG] 2017-07-12T07:22:50.318Z    e8984d42-66d2-11e7-a7af-510b0421530a    Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.ListDocuments'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG] 2017-07-12T07:22:50.336Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG] 2017-07-12T07:22:50.336Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG] 2017-07-12T07:22:50.337Z    e8984d42-66d2-11e7-a7af-510b0421530a    Calculating signature using v4 auth.
[DEBUG] 2017-07-12T07:22:50.337Z    e8984d42-66d2-11e7-a7af-510b0421530a    CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.ListDocuments

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
769c5616cb57fca6e74861906824d48410e9a1c6052187a8c2afaeda27fb55b6
[DEBUG] 2017-07-12T07:22:50.337Z    e8984d42-66d2-11e7-a7af-510b0421530a    StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
78f369959524afc6a18bbaba60b947d6c02a25f147319e2cbbf673cb7ab4b987
[DEBUG] 2017-07-12T07:22:50.337Z    e8984d42-66d2-11e7-a7af-510b0421530a    Signature:
dabe509b02c447b3d3612fc0837ba6ccc14c0fafa6e4be6518328e97a3b83840
[DEBUG] 2017-07-12T07:22:50.337Z    e8984d42-66d2-11e7-a7af-510b0421530a    Sending http request: <PreparedRequest [POST]>
[DEBUG] 2017-07-12T07:22:50.357Z    e8984d42-66d2-11e7-a7af-510b0421530a    "POST / HTTP/1.1" 200 167
[DEBUG] 2017-07-12T07:22:50.357Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response headers: {'x-amzn-requestid': 'e96e3ab5-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '167', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG] 2017-07-12T07:22:50.357Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response body:
{
    "DocumentIdentifiers": [
        {
            "DocumentType": "Command",
            "DocumentVersion": "1",
            "Name": "ASGLogBackup",
            "Owner": "350785128185",
            "PlatformTypes": [
                "Linux"
            ],
            "SchemaVersion": "1.2"
        }
    ]
}

[DEBUG] 2017-07-12T07:22:50.358Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG] 2017-07-12T07:22:50.358Z    e8984d42-66d2-11e7-a7af-510b0421530a    No retry needed.
[DEBUG] 2017-07-12T07:22:50.358Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event before-parameter-build.ssm.SendCommand: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG] 2017-07-12T07:22:50.359Z    e8984d42-66d2-11e7-a7af-510b0421530a    Making request for OperationModel(name=SendCommand) (verify_ssl=True) with params: {'body': '{"DocumentName": "ASGLogBackup", "InstanceIds": ["i-057beff453ba8887e"], "TimeoutSeconds": 120}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.SendCommand'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG] 2017-07-12T07:22:50.376Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event request-created.ssm.SendCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG] 2017-07-12T07:22:50.376Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event choose-signer.ssm.SendCommand: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG] 2017-07-12T07:22:50.377Z    e8984d42-66d2-11e7-a7af-510b0421530a    Calculating signature using v4 auth.
[DEBUG] 2017-07-12T07:22:50.377Z    e8984d42-66d2-11e7-a7af-510b0421530a    CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.SendCommand

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
f2237f6810448b76d44e17fd66b83b1b4371df545e6117e7f66eebb311631f28
[DEBUG] 2017-07-12T07:22:50.377Z    e8984d42-66d2-11e7-a7af-510b0421530a    StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
17b7fd589cca1a6a1548ff529c6ec0884756f72920c6cf5d79e9cf8b0d55c76b
[DEBUG] 2017-07-12T07:22:50.377Z    e8984d42-66d2-11e7-a7af-510b0421530a    Signature:
1cf64492c0ebe9ddcd021fa19fbcf5caeffe46284e55f8b853242c8b1ae8c9a1
[DEBUG] 2017-07-12T07:22:50.377Z    e8984d42-66d2-11e7-a7af-510b0421530a    Sending http request: <PreparedRequest [POST]>
[DEBUG] 2017-07-12T07:22:50.543Z    e8984d42-66d2-11e7-a7af-510b0421530a    "POST / HTTP/1.1" 200 528
[DEBUG] 2017-07-12T07:22:50.557Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response headers: {'x-amzn-requestid': 'e9745538-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '528', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG] 2017-07-12T07:22:50.557Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response body:
{
    "Command": {
        "CommandId": "0ece375e-a643-4474-9642-81185bd7ff4a",
        "Comment": "",
        "CompletedCount": 0,
        "DocumentName": "ASGLogBackup",
        "ErrorCount": 0,
        "ExpiresAfter": 1499847890.54,
        "InstanceIds": [
            "i-057beff453ba8887e"
        ],
        "MaxConcurrency": "50",
        "MaxErrors": "0",
        "NotificationConfig": {
            "NotificationArn": "",
            "NotificationEvents": [],
            "NotificationType": ""
        },
        "OutputS3BucketName": "",
        "OutputS3KeyPrefix": "",
        "Parameters": {},
        "RequestedDateTime": 1499844170.54,
        "ServiceRole": "",
        "Status": "Pending",
        "StatusDetails": "Pending",
        "TargetCount": 1,
        "Targets": []
    }
}

[DEBUG] 2017-07-12T07:22:50.558Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event needs-retry.ssm.SendCommand: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG] 2017-07-12T07:22:50.576Z    e8984d42-66d2-11e7-a7af-510b0421530a    No retry needed.
[INFO]  2017-07-12T07:22:50.576Z    e8984d42-66d2-11e7-a7af-510b0421530a    Command sent: {u'Command': {u'Comment': u'', u'Status': u'Pending', u'MaxErrors': u'0', u'Parameters': {}, u'ExpiresAfter': datetime.datetime(2017, 7, 12, 8, 24, 50, 540000, tzinfo=tzlocal()), u'ServiceRole': u'', u'DocumentName': u'ASGLogBackup', u'TargetCount': 1, u'OutputS3BucketName': u'', u'NotificationConfig': {u'NotificationArn': u'', u'NotificationEvents': [], u'NotificationType': u''}, u'CompletedCount': 0, u'Targets': [], u'StatusDetails': u'Pending', u'ErrorCount': 0, u'OutputS3KeyPrefix': u'', u'RequestedDateTime': datetime.datetime(2017, 7, 12, 7, 22, 50, 540000, tzinfo=tzlocal()), u'CommandId': u'0ece375e-a643-4474-9642-81185bd7ff4a', u'InstanceIds': [u'i-057beff453ba8887e'], u'MaxConcurrency': u'50'}, 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'e9745538-66d2-11e7-bf0c-c5a737f98d7b', 'HTTPHeaders': {'x-amzn-requestid': 'e9745538-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '528', 'content-type': 'application/x-amz-json-1.1'}}}
[DEBUG] 2017-07-12T07:22:50.577Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event before-parameter-build.ssm.ListCommandInvocations: calling handler <function generate_idempotent_uuid at 0x7faf2e0e4d70>
[DEBUG] 2017-07-12T07:22:50.577Z    e8984d42-66d2-11e7-a7af-510b0421530a    Making request for OperationModel(name=ListCommandInvocations) (verify_ssl=True) with params: {'body': '{"InstanceId": "i-057beff453ba8887e", "CommandId": "0ece375e-a643-4474-9642-81185bd7ff4a", "Details": false}', 'url': u'https://ssm.us-east-1.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.4.4 Python/2.7.12 Linux/4.4.51-40.60.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.5.52', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'AmazonSSM.ListCommandInvocations'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7faf2de34390>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
[DEBUG] 2017-07-12T07:22:50.596Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event request-created.ssm.ListCommandInvocations: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf2de34290>>
[DEBUG] 2017-07-12T07:22:50.597Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event choose-signer.ssm.ListCommandInvocations: calling handler <function set_operation_specific_signer at 0x7faf2e0e4c80>
[DEBUG] 2017-07-12T07:22:50.597Z    e8984d42-66d2-11e7-a7af-510b0421530a    Calculating signature using v4 auth.
[DEBUG] 2017-07-12T07:22:50.597Z    e8984d42-66d2-11e7-a7af-510b0421530a    CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.us-east-1.amazonaws.com
x-amz-date:20170712T072250Z
x-amz-security-token:FQoDYXdzELn//////////wEaDHM/oDkHT6rjH9WwOSLmAeFeEyc0+90TM2MhGBO5LPWOa1/fhPFpanPE0aLyIOsVnuvk8EbiGObL/t2zL8YMeVWJNsNB4F1StC9Z7aiUF35Q6viWQSEsk/1T3fpuwRp9r8MWPLJZRlLjZecl4OmYcC3a1IzMn4oEN1gJxRJaZCckizyr7m9Mkm493f0pB9M6Pa7QDz3D5+YCy0qu3+NsKApSgEobc2HQnXtn1GE3/zFrjLxKTsWnSFuTEuLGrSsVWoBs5b4EKTJHnBSRcpL3a/A7++CVG/zxVHt8QSjQ/Cb4GQZx/jyQHJZCP7uqnge2DBKB2/QuKMmcl8sF
x-amz-target:AmazonSSM.ListCommandInvocations

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
06321604afedcd5513b2df3a174aad8faf73c23978da2c8fa03ff26689ddbaa0
[DEBUG] 2017-07-12T07:22:50.597Z    e8984d42-66d2-11e7-a7af-510b0421530a    StringToSign:
AWS4-HMAC-SHA256
20170712T072250Z
20170712/us-east-1/ssm/aws4_request
aeddc98a9eb7a4eaec8aa55d7eefe079d6f56c45766155b52abf22bdc4818c38
[DEBUG] 2017-07-12T07:22:50.597Z    e8984d42-66d2-11e7-a7af-510b0421530a    Signature:
e5fc7fac6dae183c6697431e16d769d2be5555680f04742f4f2783c8ec5abe7c
[DEBUG] 2017-07-12T07:22:50.598Z    e8984d42-66d2-11e7-a7af-510b0421530a    Sending http request: <PreparedRequest [POST]>
[DEBUG] 2017-07-12T07:22:50.631Z    e8984d42-66d2-11e7-a7af-510b0421530a    "POST / HTTP/1.1" 200 25
[DEBUG] 2017-07-12T07:22:50.631Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response headers: {'x-amzn-requestid': 'e995e711-66d2-11e7-bf0c-c5a737f98d7b', 'date': 'Wed, 12 Jul 2017 07:22:49 GMT', 'content-length': '25', 'content-type': 'application/x-amz-json-1.1'}
[DEBUG] 2017-07-12T07:22:50.631Z    e8984d42-66d2-11e7-a7af-510b0421530a    Response body:
{
    "CommandInvocations": []
}

[DEBUG] 2017-07-12T07:22:50.632Z    e8984d42-66d2-11e7-a7af-510b0421530a    Event needs-retry.ssm.ListCommandInvocations: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf2de0f150>
[DEBUG] 2017-07-12T07:22:50.632Z    e8984d42-66d2-11e7-a7af-510b0421530a    No retry needed.
[ERROR] 2017-07-12T07:22:50.632Z    e8984d42-66d2-11e7-a7af-510b0421530a    Error: list index out of range
END RequestId: e8984d42-66d2-11e7-a7af-510b0421530a
REPORT RequestId: e8984d42-66d2-11e7-a7af-510b0421530a  Duration: 561.05 ms Billed Duration: 600 ms Memory Size: 128 MB Max Memory Used: 40 MB

I couldn't find any more details to troubleshoot...

ghost commented 6 years ago

@diegonat @ovalba I am facing the same issue, is there an update to this script ?

binoykrishnamedhi commented 6 years ago

Change the function check_command as below :

def check_command(command_id, instance_id):
    timewait = 1
    time.sleep(timewait)
    timewait += timewait
    while True:
        response_iterator = ssm_client.list_command_invocations(
            CommandId = command_id, 
            InstanceId = instance_id, 
            Details=False
            )
        if check_response(response_iterator):
            response_iterator_status = response_iterator['CommandInvocations'][0]['Status']
            if response_iterator_status != 'Pending':
                if response_iterator_status == 'InProgress' or response_iterator_status == 'Success':
                    logging.info( "Status: %s", response_iterator_status)
                    return True
                else:
                    logging.error("ERROR: status: %s", response_iterator)
                    return False
diegonat commented 6 years ago

sorry for the late response.

Can you try the following and let me know?

def check_command(command_id, instance_id):
    timewait = 1
    time.sleep(timewait)
    while True:
        response_iterator = ssm_client.list_command_invocations(
            CommandId = command_id,
            InstanceId = instance_id,
            Details=False
            )
        if check_response(response_iterator):
            try:
                response_iterator_status = response_iterator['CommandInvocations'][0]['Status']
                logging.info( "CommandInvocations Status: %s", response_iterator['CommandInvocations'])
                if response_iterator_status != 'Pending':
                    if response_iterator_status == 'InProgress' or response_iterator_status == 'Success':
                        logging.info( "Status: %s", response_iterator_status)
                        return True
                    else:
                        logging.error("ERROR: status: %s", response_iterator)
                        return False
            except Exception as e:
                logging.error("ERROR: %s", str(e))
                logging.error("ERROR: status: %s", response_iterator)
                return False
        time.sleep(timewait)
        timewait += timewait
monishamsingh commented 6 years ago

Also, you can try the below:

{ "schemaVersion": "1.2", "description": "Backup logs to S3", "parameters": {}, "runtimeConfig": { "aws:runShellScript": { "properties": [ { "id": "0.aws:runShellScript", "runCommand": [ "", "ASGNAME='xxxxx'", "LIFECYCLEHOOKNAME='xxx'", "BACKUPDIRECTORY='/var/log/'", "S3BUCKET='xxxxx'", "SNSTARGET='arn:aws:sns:ap-south-1:1234567:xxxxx'",
"INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)", "REGION=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)", "REGION='ap-south-1'", "HOOKRESULT='CONTINUE'", "MESSAGE='test_asg'", "", "tar -vcf /tmp/${INSTANCEID}.tar -P $BACKUPDIRECTORY &> /tmp/backup", "if [ $? -ne 0 ]", "then", " MESSAGE=$(cat /tmp/backup)", "else", " aws s3 cp /tmp/${INSTANCEID}.tar s3://${S3BUCKET}/${INSTANCEID}/ &> /tmp/backup", " MESSAGE=$(cat /tmp/backup)", "fi", "", "aws sns publish --subject 'ASG Backup' --message \"$MESSAGE\" --target-arn ${SNSTARGET} --region ${REGION}", "aws autoscaling complete-lifecycle-action --lifecycle-hook-name ${LIFECYCLEHOOKNAME} --auto-scaling-group-name ${ASGNAME} --lifecycle-action-result ${HOOKRESULT} --instance-id ${INSTANCEID} --region ${REGION}" ] } ] } } }

monishamsingh commented 6 years ago

If u have multiple Auto-scaling groups,

{ "schemaVersion": "1.2", "description": "Backup logs to S3", "parameters": {}, "runtimeConfig": { "aws:runShellScript": { "properties": [ { "id": "0.aws:runShellScript", "runCommand": [ "", "INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)", "ASGNAME=aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCEID | grep AutoScalingGroupName | cut -d: -f2 | tr -d ',' | sed \"s\/^[ \\t]*\/\/\" | tr -d '\"'", "LIFECYCLEHOOKNAME='xxxxx'", "BACKUPDIRECTORY='/var/log/'", "S3BUCKET='xxxxx'", "SNSTARGET='arn:aws:sns:ap-south-1:1234567:xxxxxx'",
"INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)", "REGION=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)", "REGION='ap-south-1'", "HOOKRESULT='CONTINUE'", "MESSAGE='xxxxx'", "", "tar -vcf /tmp/${INSTANCEID}.tar -P $BACKUPDIRECTORY &> /tmp/backup", "if [ $? -ne 0 ]", "then", " MESSAGE=$(cat /tmp/backup)", "else", " aws s3 cp /tmp/${INSTANCEID}.tar s3://${S3BUCKET}/${INSTANCEID}/ &> /tmp/backup", " MESSAGE=$(cat /tmp/backup)", "fi", "", "aws sns publish --subject 'ASG Backup' --message \"$MESSAGE\" --target-arn ${SNSTARGET} --region ${REGION}", "aws autoscaling complete-lifecycle-action --lifecycle-hook-name ${LIFECYCLEHOOKNAME} --auto-scaling-group-name ${ASGNAME} --lifecycle-action-result ${HOOKRESULT} --instance-id ${INSTANCEID} --region ${REGION}" ] } ] } } }

ghost commented 6 years ago

You need to add the region to boto3 client commands:

  1. Declare a global variable for aws region 
    import os
AWS_REGION = os.environ['AWS_REGION']
  2. Update boto3 client to use region 
    line 8: ssm_client = boto3.client("ssm", region_name=AWS_REGION)
line 96: asg_client = boto3.client('autoscaling', region_name=AWS_REGION)
matrix200 commented 6 years ago

I am also having this error, so what is the ultimate fix?

diegonat commented 6 years ago

I'd kindly ask you to use the code above and see if it works

matrix200 commented 6 years ago

I am just a little confused from various code snippets. Are you referring to the one you posted on April the 11th?

matrix200 commented 6 years ago

OK so after using your code from April 11th, this error is gone, but I still get nothing in the bucket. Here are the logs: 15:17:46 [DEBUG] 2018-08-30T15:17:46.992Z Loading variable profile from defaults. [DEBUG] 2018-08-30T15:17:46.992Z Loading variable profile from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.992Z Loading variable config_file from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.992Z Loading variable credentials_file from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.992Z Loading variable data_path from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable region from environment with value 'us-west-2'. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults. [DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable ca_bundle from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable api_versions from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable credentials_file from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable config_file from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.993Z Loading variable profile from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.994Z Loading variable metadata_service_timeout from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.994Z Loading variable profile from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.994Z Loading variable metadata_service_num_attempts from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.998Z Loading variable profile from defaults. 15:17:46 [DEBUG] 2018-08-30T15:17:46.998Z Looking for credentials via: env 15:17:46 [INFO] 2018-08-30T15:17:46.998Z Found credentials in environment variables. 15:17:47 [DEBUG] 2018-08-30T15:17:47.0Z Loading JSON file: /var/runtime/botocore/data/endpoints.json 15:17:47 [DEBUG] 2018-08-30T15:17:47.29Z Loading variable profile from defaults. 15:17:47 [DEBUG] 2018-08-30T15:17:47.29Z Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f78a5e53cf8> 15:17:47 [DEBUG] 2018-08-30T15:17:47.211Z Loading JSON file: /var/runtime/botocore/data/ssm/2014-11-06/service-2.json 15:17:47 [DEBUG] 2018-08-30T15:17:47.290Z Event creating-client-class.ssm: calling handler <function add_generate_presigned_url at 0x7f78a5e8e320> 15:17:47 [DEBUG] 2018-08-30T15:17:47.291Z The s3 config key is not a dictionary type, ignoring its value of: None 15:17:47 [DEBUG] 2018-08-30T15:17:47.293Z Setting ssm timeout as (60, 60) 15:17:47 [DEBUG] 2018-08-30T15:17:47.294Z Loading JSON file: /var/runtime/botocore/data/_retry.json 15:17:47 [DEBUG] 2018-08-30T15:17:47.297Z Registering retry handlers for service: ssm 15:17:47 START RequestId: d95427a2-ac67-11e8-bfed-bdb957ac3432 Version: $LATEST 15:17:47 [INFO] 2018-08-30T15:17:47.298Z d95427a2-ac67-11e8-bfed-bdb957ac3432 {"account": "215691954377", "region": "us-west-2", "detail": {"LifecycleHookName": "ASGBackup", "AutoScalingGroupName": "multicastingNewServer", "LifecycleActionToken": "4b215a7d-f41b-4050-9ec0-b80230e88588", "LifecycleTransition": "autoscaling:EC2_INSTANCE_TERMINATING", "EC2InstanceId": "i-071f6ae185996bc6f"}, "detail-type": "EC 15:17:47 [DEBUG] 2018-08-30T15:17:47.298Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50> 15:17:47 [DEBUG] 2018-08-30T15:17:47.299Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119-44.140.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 15:17:47 [DEBUG] 2018-08-30T15:17:47.307Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>> 15:17:47 [DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60> 15:17:47 [DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth. 15:17:47 [DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151747Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A 15:17:47 [DEBUG] 2018-08-30T15:17:47.308Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151747Z 20180830/us-west-2/ssm/aws4_request 7f18f52c48499da58730fcb7881f155261bd1435451672a6f3804ec57a6bc433 15:17:47 [DEBUG] 2018-08-30T15:17:47.309Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 7f32c91638e59cc92792c6498357803ded2e5ede7502cedc8ceb93260282569e 15:17:47 [DEBUG] 2018-08-30T15:17:47.337Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]> 15:17:47 [INFO] 2018-08-30T15:17:47.338Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Starting new HTTPS connection (1): ssm.us-west-2.amazonaws.com 15:17:47 [DEBUG] 2018-08-30T15:17:47.563Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 201 15:17:47 [DEBUG] 2018-08-30T15:17:47.564Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': '1f875bf1-b251-448b-a3e0-e521f139449b', 'date': 'Thu, 30 Aug 2018 15:17:47 GMT', 'content-length': '201', 'content-type': 'application/x-amz-json-1.1'} 15:17:47 [DEBUG] 2018-08-30T15:17:47.564Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"DocumentIdentifiers":[{"DocumentFormat":"JSON","DocumentType":"Command","DocumentVersion":"1","Name":"ASGLogBackup","Owner":"215691954377","PlatformTypes":["Linux"],"SchemaVersion":"1.2","Tags":[]}]} 15:17:47 [DEBUG] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10> 15:17:47 [DEBUG] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed. 15:17:47 [INFO] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Documents list: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'Tags': [], u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'215691954377', u'SchemaVersion': u'1.2', u'DocumentFormat': u'JSON'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '1f8 15:17:47 [INFO] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Documents exists: {u'DocumentIdentifiers': [{u'Name': u'ASGLogBackup', u'Tags': [], u'PlatformTypes': [u'Linux'], u'DocumentVersion': u'1', u'DocumentType': u'Command', u'Owner': u'215691954377', u'SchemaVersion': u'1.2', u'DocumentFormat': u'JSON'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '1 15:17:47 [DEBUG] 2018-08-30T15:17:47.565Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.ListDocuments: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50> 15:17:47 [DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=ListDocuments) (verify_ssl=True) with params: {'body': '{"DocumentFilterList": [{"key": "Name", "value": "ASGLogBackup"}]}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119-44.140.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 15:17:47 [DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.ListDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>> 15:17:47 [DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.ListDocuments: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60> 15:17:47 [DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth. 15:17:47 [DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151747Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A 15:17:47 [DEBUG] 2018-08-30T15:17:47.566Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151747Z 20180830/us-west-2/ssm/aws4_request 7f18f52c48499da58730fcb7881f155261bd1435451672a6f3804ec57a6bc433 15:17:47 [DEBUG] 2018-08-30T15:17:47.567Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 7f32c91638e59cc92792c6498357803ded2e5ede7502cedc8ceb93260282569e 15:17:47 [DEBUG] 2018-08-30T15:17:47.567Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]> 15:17:47 [DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 201 15:17:47 [DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': 'c7ba175a-7f5f-467a-a195-971a1ada437a', 'date': 'Thu, 30 Aug 2018 15:17:47 GMT', 'content-length': '201', 'content-type': 'application/x-amz-json-1.1'} 15:17:47 [DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"DocumentIdentifiers":[{"DocumentFormat":"JSON","DocumentType":"Command","DocumentVersion":"1","Name":"ASGLogBackup","Owner":"215691954377","PlatformTypes":["Linux"],"SchemaVersion":"1.2","Tags":[]}]} 15:17:47 [DEBUG] 2018-08-30T15:17:47.628Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.ListDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10> 15:17:47 [DEBUG] 2018-08-30T15:17:47.629Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed. 15:17:47 [DEBUG] 2018-08-30T15:17:47.629Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.SendCommand: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50> 15:17:47 [DEBUG] 2018-08-30T15:17:47.630Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=SendCommand) (verify_ssl=True) with params: {'body': '{"DocumentName": "ASGLogBackup", "InstanceIds": ["i-071f6ae185996bc6f"], "TimeoutSeconds": 120}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119-44.140.amzn1.x86_64 exe 15:17:47 [DEBUG] 2018-08-30T15:17:47.630Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.SendCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>> 15:17:47 [DEBUG] 2018-08-30T15:17:47.630Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.SendCommand: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60> 15:17:47 [DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth. 15:17:47 [DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151747Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A 15:17:47 [DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151747Z 20180830/us-west-2/ssm/aws4_request 839c71b66e35d2072a1b08f639b7e3eb3888bd325114c8301d3e40db2773a071 15:17:47 [DEBUG] 2018-08-30T15:17:47.689Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 156c189eb80f4ee393375987eaa9209d196a2143332141a5bfeec85886842f43 15:17:47 [DEBUG] 2018-08-30T15:17:47.690Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]> 15:17:47 [DEBUG] 2018-08-30T15:17:47.849Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 684 15:17:47 [DEBUG] 2018-08-30T15:17:47.849Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': 'c4a72f3b-6b89-4c34-939b-f59637f9b408', 'date': 'Thu, 30 Aug 2018 15:17:47 GMT', 'content-length': '684', 'content-type': 'application/x-amz-json-1.1'} 15:17:47 [DEBUG] 2018-08-30T15:17:47.849Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"Command":{"CloudWatchOutputConfig":{"CloudWatchLogGroupName":"","CloudWatchOutputEnabled":false},"CommandId":"653f9d43-0a30-4204-9d4c-865d0a481983","Comment":"","CompletedCount":0,"DeliveryTimedOutCount":0,"DocumentName":"ASGLogBackup","DocumentVersion":"","ErrorCount":0,"ExpiresAfter":1.535645987832E9,"Instance 15:17:47 [DEBUG] 2018-08-30T15:17:47.867Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.SendCommand: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10> 15:17:47 [DEBUG] 2018-08-30T15:17:47.868Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed. 15:17:47 [INFO] 2018-08-30T15:17:47.868Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Command sent: {u'Command': {u'MaxErrors': u'0', u'Parameters': {}, u'DocumentName': u'ASGLogBackup', u'OutputS3BucketName': u'', u'OutputS3KeyPrefix': u'', u'StatusDetails': u'Pending', u'RequestedDateTime': datetime.datetime(2018, 8, 30, 15, 17, 47, 832000, tzinfo=tzlocal()), u'Status': u'Pending', u'TargetCount': 1, u'Notificat 15:17:48 [DEBUG] 2018-08-30T15:17:48.869Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event before-parameter-build.ssm.ListCommandInvocations: calling handler <function generate_idempotent_uuid at 0x7f78a5e53f50> 15:17:48 [DEBUG] 2018-08-30T15:17:48.870Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Making request for OperationModel(name=ListCommandInvocations) (verify_ssl=True) with params: {'body': '{"InstanceId": "i-071f6ae185996bc6f", "CommandId": "653f9d43-0a30-4204-9d4c-865d0a481983", "Details": false}', 'url': u'https://ssm.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'Boto3/1.7.74 Python/2.7.12 Linux/4.9.119 15:17:48 [DEBUG] 2018-08-30T15:17:48.870Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event request-created.ssm.ListCommandInvocations: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f78a5ae37d0>> 15:17:48 [DEBUG] 2018-08-30T15:17:48.870Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event choose-signer.ssm.ListCommandInvocations: calling handler <function set_operation_specific_signer at 0x7f78a5e53e60>  15:17:48 [DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Calculating signature using v4 auth. 15:17:48 [DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CanonicalRequest: POST / content-type:application/x-amz-json-1.1 host:ssm.us-west-2.amazonaws.com x-amz-date:20180830T151748Z x-amz-security-token:FQoGZXIvYXdzENn//////////wEaDHrK2D0CtjL9b5VYqiLmAYD8ohjS2Os5TKDGvkgreVNovs7NAlgYX/ca93BzkQwKBOc3E2z9ZK9RJWAVY6FeqKuJL/7cilFXVd+gpcOtIXBRDUX5xHg4dXJSKmA2LAIJgkOH5dqudV2UgXq7rF7Dz2Q20A 15:17:48 [DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 StringToSign: AWS4-HMAC-SHA256 20180830T151748Z 20180830/us-west-2/ssm/aws4_request 6089f8a64d5097bd60ea9c671d2fba8c8cd1aa2063b6c180f2b7c0a4941bd07a 15:17:48 [DEBUG] 2018-08-30T15:17:48.871Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Signature: 5860ada82ebd6599a535605ad545111abfb53455e75f2b69eb37d7f31fecf031 15:17:48 [DEBUG] 2018-08-30T15:17:48.888Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Sending http request: <PreparedRequest [POST]> 15:17:48 [DEBUG] 2018-08-30T15:17:48.906Z d95427a2-ac67-11e8-bfed-bdb957ac3432 "POST / HTTP/1.1" 200 540 15:17:48 [DEBUG] 2018-08-30T15:17:48.907Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response headers: {'x-amzn-requestid': '6fa68de8-f293-4754-8217-c15230edda82', 'date': 'Thu, 30 Aug 2018 15:17:48 GMT', 'content-length': '540', 'content-type': 'application/x-amz-json-1.1'} 15:17:48 [DEBUG] 2018-08-30T15:17:48.907Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Response body: {"CommandInvocations":[{"CloudWatchOutputConfig":{"CloudWatchLogGroupName":"","CloudWatchOutputEnabled":false},"CommandId":"653f9d43-0a30-4204-9d4c-865d0a481983","CommandPlugins":[],"Comment":"","DocumentName":"ASGLogBackup","DocumentVersion":"","InstanceId":"i-071f6ae185996bc6f","InstanceName":"","NotificationCon 15:17:48 [DEBUG] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Event needs-retry.ssm.ListCommandInvocations: calling handler <botocore.retryhandler.RetryHandler object at 0x7f78a5a99a10> 15:17:48 [DEBUG] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 No retry needed. 15:17:48 [INFO] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 CommandInvocations Status: [{u'Comment': u'', u'Status': u'InProgress', u'CommandPlugins': [], u'ServiceRole': u'', u'CloudWatchOutputConfig': {u'CloudWatchLogGroupName': u'', u'CloudWatchOutputEnabled': False}, u'InstanceId': u'i-071f6ae185996bc6f', u'DocumentName': u'ASGLogBackup', u'NotificationConfig': {u'NotificationArn': u' 15:17:48 [INFO] 2018-08-30T15:17:48.929Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Status: InProgress 15:17:48 [INFO] 2018-08-30T15:17:48.930Z d95427a2-ac67-11e8-bfed-bdb957ac3432 Lambda executed correctly 15:17:48 END RequestId: d95427a2-ac67-11e8-bfed-bdb957ac3432 15:17:48 REPORT RequestId: d95427a2-ac67-11e8-bfed-bdb957ac3432 Duration: 1631.98 ms Billed Duration: 1700 ms Memory Size: 128 MB Max Memory Used: 48 MB

The only thing that kind of looks like an error is The s3 config key is not a dictionary type, ignoring its value of: None Also no tar files are created on the server while the script runs. What am I missing?

diegonat commented 6 years ago

it seems that the commend was running. Did you check the logs of the SSM? Do you see any error? If you look at [{u'Comment': u'', u'Status': u'InProgress', u'Command.... the status is in progress. Does the instance has the permission to write to S3?

matrix200 commented 6 years ago

Not sure about the S3, I thought no permission was necessary as long as the account itself has write permission? But the problem is obviously before S3 as the files aren't even getting created on the instance (the tar archives). You are right, I have got to go over SSM logs. Will do in a few.

matrix200 commented 6 years ago

Here is output of the command in SSM logs: % Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 19 100 19 0 0 26874 0 --:--:-- --:--:-- --:--:-- 19000

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 10 100 10 0 0 21097 0 --:--:-- --:--:-- --:--:-- 10000

/var/lib/amazon/ssm/i-071f6ae185996bc6f/document/orchestration/653f9d43-0a30-4204-9d4c-865d0a481983/awsrunShellScript/0.awsrunShellScript/_script.sh: 8: /var/lib/amazon/ssm/i-071f6ae185996bc6f/document/orchestration/653f9d43-0a30-4204-9d4c-865d0a481983/awsrunShellScript/0.awsrunShellScript/_script.sh: Bad substitution

failed to run commands: exit status 2

diegonat commented 6 years ago

this error is not related to the lambda_backup.py script. It is an error with the document that is ran within the instance. Try to run it in an instance that is not part of ASG to troubleshoot the issue. Please open another issue.

akhileshjoshi2 commented 6 years ago

getting the below error

[DEBUG] 2018-10-23T09:41:35.581Z d37829b0-d6a7-11e8-b0eb-f5d458a80990 The s3 config key is not a dictionary type, ignoring its value of: None

akhileshjoshi2 commented 6 years ago

I have redhat servers. Have given the s3 access to the servers. Have installed the aws cli on the servers. Have installed the ssm-agent too. I can upload manually from the servers but while running through lambda and following the steps im getting this error.

[DEBUG] 2018-10-23T09:41:35.581Z d37829b0-d6a7-11e8-b0eb-f5d458a80990 The s3 config key is not a dictionary type, ignoring its value of: None

#####################################################################

BalaValluri commented 5 years ago

Any further luck with this?.

I am also facing the same issue, Cloudwatch log says "Lambda executed correctly". But the files are not being copied to S3. The files are being copied from the same instance while tried manually (when ran the copy command within the instance).

matrix200 commented 5 years ago

I ended up not sending the logs to S3 and instead just keep sending them to another ec2 instance which is always up.

BalaValluri commented 5 years ago

It's working fine for me and copying the files to S3 bucket using the trick at https://github.com/aws/amazon-ssm-agent/issues/46#issuecomment-296515703