search

aws-ia / taskcat

Test all the CloudFormation things! (with TaskCat)
https://aws-ia.github.io/taskcat/
Apache License 2.0
1.17k stars 213 forks source link

Taskcat fails with templates using macros #250

Closed matteofigus closed 5 years ago

matteofigus commented 5 years ago

Describe the bug When using macros (in my case AWS::Serverless) taskcat exits with the following error:

Traceback (most recent call last):
  File "/Users/mfigus/.pyenv/versions/3.7.1/bin/taskcat", line 101, in <module>
    main()
  File "/Users/mfigus/.pyenv/versions/3.7.1/bin/taskcat", line 88, in main
    testdata = tcat_instance.stackcreate(taskcat_cfg, test_list, args.stack_prefix)
  File "/Users/mfigus/.local/lib/python3.7/site-packages/taskcat/stacker.py", line 728, in stackcreate
    Tags=self.tags
  File "/Users/mfigus/.local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/mfigus/.local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InsufficientCapabilitiesException: An error occurred (InsufficientCapabilitiesException) when calling the CreateStack operation: Requires capabilities : [CAPABILITY_AUTO_EXPAND]

To Reproduce Steps to reproduce the behavior: Here is my template:

Expand

```yml AWSTemplateFormatVersion: '2010-09-09' Globals: Api: Cors: AllowHeaders: '''Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token''' AllowMethods: '''*''' AllowOrigin: '''*''' Function: Environment: Variables: API_GATEWAY: 'Fn::Sub': 'https://${RestApi}.execute-api.${AWS::Region}.amazonaws.com/PROD' COGNITO_IDENTITY_POOL: Ref: CognitoIdentityPool COLLECTION_ID: Ref: CollectionId FROM_BUCKET: rekognition-engagement-meter REGION: Ref: 'AWS::Region' TO_BUCKET: Ref: WebUIBucket Outputs: url: Description: Engagement Meter URL Value: 'Fn::Sub': 'https://${WebUIBucket}.s3.amazonaws.com/index.html' Parameters: CollectionId: AllowedPattern: '^[a-zA-Z0-9_]*$' Default: RekogDemo Type: String Resources: ApiGatewayInvokeRole: Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRoleWithWebIdentity' Effect: Allow Principal: Federated: - cognito-identity.amazonaws.com ManagedPolicyArns: - 'arn:aws:iam::aws:policy/AmazonAPIGatewayInvokeFullAccess' Type: 'AWS::IAM::Role' CognitoIdentityPool: Properties: AllowUnauthenticatedIdentities: true IdentityPoolName: 'Fn::Sub': 'RekogIdentityPool${CollectionId}' Type: 'AWS::Cognito::IdentityPool' CognitoIdentityPoolRole: Properties: IdentityPoolId: Ref: CognitoIdentityPool Roles: authenticated: 'Fn::GetAtt': - ApiGatewayInvokeRole - Arn unauthenticated: 'Fn::GetAtt': - ApiGatewayInvokeRole - Arn Type: 'AWS::Cognito::IdentityPoolRoleAttachment' CustomResourceRole: Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - lambda.amazonaws.com Path: / Policies: - PolicyDocument: Statement: - Action: 's3:*' Effect: Allow Resource: '*' PolicyName: RekogDemo-setup-S3-fc - PolicyDocument: Statement: - Action: 'rekognition:*' Effect: Allow Resource: '*' PolicyName: RekogDemo-createRekColl - PolicyDocument: Statement: - Action: 'logs:*' Effect: Allow Resource: 'arn:aws:logs:*:*:*' PolicyName: RekogDemo-cloudwatch-logs Type: 'AWS::IAM::Role' DbReadRole: Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - apigateway.amazonaws.com ManagedPolicyArns: - 'arn:aws:iam::aws:policy/AmazonDynamoDBReadOnlyAccess' Type: 'AWS::IAM::Role' DbWriteRole: Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - apigateway.amazonaws.com ManagedPolicyArns: - 'arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess' Type: 'AWS::IAM::Role' FacesDynamoTable: Properties: AttributeDefinitions: - AttributeName: CollectionId AttributeType: S - AttributeName: ExternalImageId AttributeType: S - AttributeName: MemberName AttributeType: S - AttributeName: JobTitle AttributeType: S BillingMode: PAY_PER_REQUEST GlobalSecondaryIndexes: - IndexName: geGSI KeySchema: - AttributeName: JobTitle KeyType: HASH - AttributeName: MemberName KeyType: RANGE Projection: NonKeyAttributes: - CollectionId - ExternalImageId ProjectionType: INCLUDE KeySchema: - AttributeName: CollectionId KeyType: HASH - AttributeName: ExternalImageId KeyType: RANGE TableName: 'Fn::Sub': 'RekogFaces${CollectionId}' Type: 'AWS::DynamoDB::Table' LambdaSetup: Properties: Description: 'Fn::Sub': 'Custom Lambda resource for the ${CollectionId} Cloudformation Stack' FunctionName: 'Fn::Sub': 'RekogDemoSetup${CollectionId}' Handler: index.handler MemorySize: 128 Role: 'Fn::GetAtt': - CustomResourceRole - Arn Runtime: nodejs8.10 Timeout: 30 InlineCode: >- module.exports=function(e){var t={};function o(n){if(t[n])return t[n].exports;var r=t[n]={i:n,l:!1,exports:{}};return e[n].call(r.exports,r,r.exports,o),r.l=!0,r.exports}return o.m=e,o.c=t,o.d=function(e,t,n){o.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:n})},o.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},o.t=function(e,t){if(1&t&&(e=o(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var n=Object.create(null);if(o.r(n),Object.defineProperty(n,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var r in e)o.d(n,r,function(t){return e[t]}.bind(null,r));return n},o.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return o.d(t,"a",t),t},o.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},o.p="",o(o.s=0)}([function(e,t,o){e.exports=o(1)},function(e,t,o){const n=o(2),r=o(3),s=o(4),c=o(7),{REGION:i}=process.env;t.handler=((e,t,o)=>{const{createCollection:l,deleteCollection:u}=r(new n.Rekognition({region:i})),{copyFiles:a,removeFiles:p,writeSettings:d}=c(new n.S3),{sendResponse:f}=s(e,t,o),g=e.RequestType;let y;"Create"===g?(console.log("Creating resources"),y=[a(),d(),l()]):"Delete"===g&&(console.log("Deleting resources"),y=[p(),u()]),Promise.all(y).then(()=>(console.log("All actions successfully performed"),f("SUCCESS",{Message:`Resources successfully ${g.toLowerCase()}d`}))).catch(e=>console.log(e)||f("FAILED"))})},function(e,t){e.exports=require("aws-sdk")},function(e,t){const{COLLECTION_ID:o}=process.env,n={CollectionId:o};e.exports=(e=>({createCollection:()=>e.createCollection(n).promise(),deleteCollection:()=>e.deleteCollection(n).promise()}))},function(e,t,o){const n=o(5),r=o(6);e.exports=((e,t,o)=>{let s;const c=e=>(clearTimeout(s),e&&(console.log("There was an error"),console.log(e)),o(e)),i=(o,s)=>{const i=JSON.stringify({Status:o,Reason:`Details: ${t.logStreamName}`,PhysicalResourceId:t.logStreamName,StackId:e.StackId,RequestId:e.RequestId,LogicalResourceId:e.LogicalResourceId,Data:s}),l={url:e.ResponseURL,body:i,method:"PUT"};return console.log(`Making HTTP request to ${e.ResponseURL}: ${i}`),(e=>{const t=Object.assign({},r.parse(e.url),{method:e.method,headers:{"Content-Type":"","Content-Length":Buffer.byteLength(e.body)}});return new Promise((o,r)=>{const s=n.request(t,e=>{const t=[];if(200!==e.statusCode)return r(e);e.setEncoding("utf8"),e.on("data",e=>t.push(e)),e.on("error",r),e.on("end",()=>o(Buffer.concat(t)))});s.write(e.body),s.end()})})(l).then(e=>(console.log(e),c())).catch(e=>c(e))};return s=setTimeout(()=>i("FAILED").then(()=>o(new Error("Function timed out"))),t.getRemainingTimeInMillis()-1e3),{sendResponse:i}})},function(e,t){e.exports=require("https")},function(e,t){e.exports=require("url")},function(e,t){const{API_GATEWAY:o,COGNITO_IDENTITY_POOL:n,FROM_BUCKET:r,REGION:s,TO_BUCKET:c}=process.env;e.exports=(e=>{const t=t=>e.listObjects(t).promise();return{copyFiles:()=>t({Bucket:r,Prefix:"static/"}).then(t=>Promise.all(t.Contents.map(t=>(t=>e.copyObject(t).promise())({ACL:"public-read",Bucket:c,CopySource:`${r}/${t.Key}`,Key:t.Key.slice("static/".length)})))),removeFiles:()=>t({Bucket:c}).then(t=>Promise.all(t.Contents.map(e=>e.Key).map(t=>(t=>e.deleteObject(t).promise())({Bucket:c,Key:t})))),writeSettings:()=>e.putObject({ACL:"public-read",Bucket:c,Key:"settings.js",Body:`window.rekognitionSettings = ${JSON.stringify({apiGateway:o,cognitoIdentityPool:n,region:s})};`}).promise()}})}]); Type: 'AWS::Serverless::Function' RekognitionInvokeRole: Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - apigateway.amazonaws.com ManagedPolicyArns: - 'arn:aws:iam::aws:policy/AmazonRekognitionFullAccess' Type: 'AWS::IAM::Role' RestApi: Name: 'Fn::Sub': 'RekogDemo-${CollectionId}' Properties: DefinitionBody: basePath: /PROD info: title: 'Fn::Sub': 'RekogDemo-${CollectionId}' version: 1 paths: /engagement: get: consumes: - application/json produces: - application/json responses: '200': description: 200 response headers: Access-Control-Allow-Headers: type: string Access-Control-Allow-Origin: type: string schema: $ref: '#/definitions/Empty' security: - sigv4: [] x-amazon-apigateway-integration: credentials: 'Fn::GetAtt': - DbReadRole - Arn httpMethod: POST passthroughBehavior: when_no_match requestTemplates: application/json: 'Fn::Sub': | { "TableName": "${SentimentDynamoTable}", "IndexName": "gsiSentiment", "KeyConditionExpression": "CollectionId = :cid", "FilterExpression": "TimeDetected >= :td", "ProjectionExpression": "Angry,Confused,Happy,Sad,Surprised", "ExpressionAttributeValues": { ":cid": { "S": "${CollectionId}" }, ":td": { "N": "$input.params().querystring.get('timeDetected')" } } } responses: default: responseParameters: method.response.header.Access-Control-Allow-Headers: '''Content-Type''' method.response.header.Access-Control-Allow-Origin: '''*''' responseTemplates: application/json: | #set($inputRoot = $input.path('$')) { "results": [ #foreach($elem in $inputRoot.Items) { "angry": "$elem.Angry.S", "confused": "$elem.Confused.S", "happy": "$elem.Happy.S", "sad": "$elem.Sad.S", "surprised": "$elem.Surprised.S" }#if($foreach.hasNext),#end #end ] } statusCode: '200' type: aws uri: 'Fn::Sub': 'arn:aws:apigateway:${AWS::Region}:dynamodb:action/Query' post: consumes: - application/json produces: - application/json responses: '200': description: 200 response headers: Access-Control-Allow-Headers: type: string Access-Control-Allow-Origin: type: string schema: $ref: '#/definitions/Empty' security: - sigv4: [] x-amazon-apigateway-integration: credentials: 'Fn::GetAtt': - DbWriteRole - Arn httpMethod: POST passthroughBehavior: when_no_match requestTemplates: application/json: 'Fn::Sub': | { "TableName": "${SentimentDynamoTable}", "Item": { "CollectionId": { "S": "${CollectionId}" }, "TimeDetected": { "N": "$input.json('$.timeDetected')" }, "Angry": { "S": "$input.json('$.angry')" }, "Confused": { "S": "$input.json('$.confused')" }, "Happy": { "S": "$input.json('$.happy')" }, "Sad": { "S": "$input.json('$.sad')" }, "Surprised": { "S": "$input.json('$.surprised')" } } } responses: default: responseParameters: method.response.header.Access-Control-Allow-Headers: '''Content-Type''' method.response.header.Access-Control-Allow-Origin: '''*''' responseTemplates: application/json: | { "ok": true } statusCode: '200' type: aws uri: 'Fn::Sub': 'arn:aws:apigateway:${AWS::Region}:dynamodb:action/PutItem' /faces/add: post: consumes: - application/json produces: - application/json responses: '200': description: 200 response headers: Access-Control-Allow-Headers: type: string Access-Control-Allow-Origin: type: string schema: $ref: '#/definitions/Empty' security: - sigv4: [] x-amazon-apigateway-integration: credentials: 'Fn::GetAtt': - RekognitionInvokeRole - Arn httpMethod: POST passthroughBehavior: when_no_match requestParameters: integration.request.header.Content-Type: '''application/x-amz-json-1.1''' integration.request.header.X-Amz-Target: '''RekognitionService.IndexFaces''' requestTemplates: application/json: 'Fn::Sub': | { "CollectionId": "${CollectionId}", "ExternalImageId": $input.json('$.externalImageId'), "Image": { "Bytes": $input.json('$.image') } } responses: default: responseParameters: method.response.header.Access-Control-Allow-Headers: '''Content-Type''' method.response.header.Access-Control-Allow-Origin: '''*''' statusCode: '200' type: aws uri: 'Fn::Sub': 'arn:aws:apigateway:${AWS::Region}:rekognition:path//' /faces/detect: post: consumes: - application/json produces: - application/json responses: '200': description: 200 response headers: Access-Control-Allow-Headers: type: string Access-Control-Allow-Origin: type: string schema: $ref: '#/definitions/Empty' security: - sigv4: [] x-amazon-apigateway-integration: credentials: 'Fn::GetAtt': - RekognitionInvokeRole - Arn httpMethod: POST passthroughBehavior: when_no_match requestParameters: integration.request.header.Content-Type: '''application/x-amz-json-1.1''' integration.request.header.X-Amz-Target: '''RekognitionService.DetectFaces''' requestTemplates: application/json: | { "Attributes": ["ALL"], "Image": { "Bytes": $input.json('$.image') } } responses: default: responseParameters: method.response.header.Access-Control-Allow-Headers: '''Content-Type''' method.response.header.Access-Control-Allow-Origin: '''*''' statusCode: '200' type: aws uri: 'Fn::Sub': 'arn:aws:apigateway:${AWS::Region}:rekognition:path//' /faces/search: post: consumes: - application/json produces: - application/json responses: '200': description: 200 response headers: Access-Control-Allow-Headers: type: string Access-Control-Allow-Origin: type: string schema: $ref: '#/definitions/Empty' security: - sigv4: [] x-amazon-apigateway-integration: credentials: 'Fn::GetAtt': - RekognitionInvokeRole - Arn httpMethod: POST passthroughBehavior: when_no_match requestParameters: integration.request.header.Content-Type: '''application/x-amz-json-1.1''' integration.request.header.X-Amz-Target: '''RekognitionService.SearchFacesByImage''' requestTemplates: application/json: 'Fn::Sub': | { "CollectionId": "${CollectionId}", "FaceMatchThreshold": 85, "MaxFaces": 5, "Image": { "Bytes": $input.json('$.image') } } responses: default: responseParameters: method.response.header.Access-Control-Allow-Headers: '''Content-Type''' method.response.header.Access-Control-Allow-Origin: '''*''' statusCode: '200' type: aws uri: 'Fn::Sub': 'arn:aws:apigateway:${AWS::Region}:rekognition:path//' /people: get: consumes: - application/json produces: - application/json responses: '200': description: 200 response headers: Access-Control-Allow-Headers: type: string Access-Control-Allow-Origin: type: string schema: $ref: '#/definitions/Empty' security: - sigv4: [] x-amazon-apigateway-integration: credentials: 'Fn::GetAtt': - DbReadRole - Arn httpMethod: POST passthroughBehavior: when_no_match requestTemplates: application/json: 'Fn::Sub': | { "TableName": "${FacesDynamoTable}", "KeyConditionExpression": "CollectionId = :cid", "ProjectionExpression": "MemberName,JobTitle,ExternalImageId", "ExpressionAttributeValues": { ":cid": { "S": "${CollectionId}" } } } responses: default: responseParameters: method.response.header.Access-Control-Allow-Headers: '''Content-Type''' method.response.header.Access-Control-Allow-Origin: '''*''' responseTemplates: application/json: | #set($inputRoot = $input.path('$')) { "people": [ #foreach($elem in $inputRoot.Items) { "externalImageId": "$elem.ExternalImageId.S", "memberName": "$elem.MemberName.S", "jobTitle": "$elem.JobTitle.S" }#if($foreach.hasNext),#end #end ] } statusCode: '200' type: aws uri: 'Fn::Sub': 'arn:aws:apigateway:${AWS::Region}:dynamodb:action/Query' post: consumes: - application/json produces: - application/json responses: '200': description: 200 response headers: Access-Control-Allow-Headers: type: string Access-Control-Allow-Origin: type: string schema: $ref: '#/definitions/Empty' security: - sigv4: [] x-amazon-apigateway-integration: credentials: 'Fn::GetAtt': - DbWriteRole - Arn httpMethod: POST passthroughBehavior: when_no_match requestTemplates: application/json: 'Fn::Sub': | { "TableName": "${FacesDynamoTable}", "Item": { "CollectionId": { "S": "${CollectionId}" }, "ExternalImageId": { "S": $input.json('$.externalImageId') }, "JobTitle": { "S": $input.json('$.jobTitle') }, "MemberName": { "S": $input.json('$.memberName') } } } responses: default: responseParameters: method.response.header.Access-Control-Allow-Headers: '''Content-Type''' method.response.header.Access-Control-Allow-Origin: '''*''' responseTemplates: application/json: | { "ok": true } statusCode: '200' type: aws uri: 'Fn::Sub': 'arn:aws:apigateway:${AWS::Region}:dynamodb:action/PutItem' securityDefinitions: sigv4: in: header name: Authorization type: apiKey x-amazon-apigateway-authtype: awsSigv4 swagger: 2 EndpointConfiguration: REGIONAL StageName: PROD Type: 'AWS::Serverless::Api' SentimentDynamoTable: Properties: AttributeDefinitions: - AttributeName: CollectionId AttributeType: S - AttributeName: TimeDetected AttributeType: 'N' BillingMode: PAY_PER_REQUEST GlobalSecondaryIndexes: - IndexName: gsiSentiment KeySchema: - AttributeName: CollectionId KeyType: HASH Projection: ProjectionType: ALL KeySchema: - AttributeName: CollectionId KeyType: HASH - AttributeName: TimeDetected KeyType: RANGE TableName: 'Fn::Sub': 'RekogSentiment${CollectionId}' Type: 'AWS::DynamoDB::Table' SetupRekognitionAndWebUI: Properties: Region: Ref: 'AWS::Region' ServiceToken: 'Fn::GetAtt': - LambdaSetup - Arn Type: 'Custom::Setup' WebUIBucket: Properties: CorsConfiguration: CorsRules: - AllowedHeaders: - '*' AllowedMethods: - GET AllowedOrigins: - '*' Id: 'Fn::Sub': 'RekogCorsRule${CollectionId}' MaxAge: 3600 Type: 'AWS::S3::Bucket' Transform: 'AWS::Serverless-2016-10-31' ```

To run it, I am using a pip installed taskcat on v0.8.24 and Python 3.7.1 with default parameters.

Expected behavior To deploy correctly with no errors

matteofigus commented 5 years ago

Verified, thanks!