Open Kasmilos opened 5 years ago
@Kasmilos thanks for report. What exactly error is reported in CloudFormation? (Above log just indicated that some error happened, but we don't know what failed exactly)
Executing with SLS_DEBUG=* does not give any additional information during the deployment
...
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - S3uploadedLambdaVersionMBSJn9aBeWIfaNaAb2fH2M18CK995rOclwVN3DlUw
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - S3uploadedCustomS31
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - S3uploadedLambdaVersionMBSJn9aBeWIfaNaAb2fH2M18CK995rOclwVN3DlUw
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - S3uploadedLambdaVersionMBSJn9aBeWIfaNaAb2fH2M18CK995rOclwVN3DlUw
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - S3uploadedCustomS31
CloudFormation - CREATE_FAILED - Custom::S3 - S3uploadedCustomS31
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - pdfcountdebug-dev
...
CloudFormation - DELETE_IN_PROGRESS - AWS::Logs::LogGroup - S3uploadedLogGroup
...
CloudFormation - DELETE_COMPLETE - AWS::Logs::LogGroup - S3uploadedLogGroup
...
It does not give any additional information about the failure reason that I can see. Because CloudFormation does a complete rollback there is no logs in CloudWatch or on the console.
The execution terminates with:
Serverless Error ---------------------------------------
ServerlessError: An error occurred: S3uploadedCustomS31 - Failed to create resource. Access Denied See details in CloudWatch Log: 2019/10/01/[$LATEST]0d3cb59b1731410db457c06299c4df35.
at provider.request.then.data (dev\serverless\lib\plugins\aws\lib\monitorStack.js:122:33)
From previous event:
at AwsDeploy.monitorStack (dev\serverless\lib\plugins\aws\lib\monitorStack.js:28:12)
at provider.request.then.cfData (dev\serverless\lib\plugins\aws\lib\updateStack.js:103:28)
From previous event:
at AwsDeploy.update (dev\serverless\lib\plugins\aws\lib\updateStack.js:103:8)
From previous event:
at AwsDeploy.BbPromise.bind.then (dev\serverless\lib\plugins\aws\lib\updateStack.js:117:35)
From previous event:
at AwsDeploy.updateStack (dev\serverless\lib\plugins\aws\lib\updateStack.js:113:33)
From previous event:
at AwsDeploy.BbPromise.bind.then (dev\serverless\lib\plugins\aws\deploy\index.js:127:39)
From previous event:
at Object.aws:deploy:deploy:updateStack [as hook] (dev\serverless\lib\plugins\aws\deploy\index.js:123:30)
at BbPromise.reduce (dev\serverless\lib\classes\PluginManager.js:505:55)
From previous event:
at PluginManager.invoke (dev\serverless\lib\classes\PluginManager.js:505:22)
at PluginManager.spawn (dev\serverless\lib\classes\PluginManager.js:525:17)
at AwsDeploy.BbPromise.bind.then (dev\serverless\lib\plugins\aws\deploy\index.js:93:48)
From previous event:
at Object.deploy:deploy [as hook] (dev\serverless\lib\plugins\aws\deploy\index.js:89:30)
at BbPromise.reduce (dev\serverless\lib\classes\PluginManager.js:505:55)
From previous event:
at PluginManager.invoke (dev\serverless\lib\classes\PluginManager.js:505:22)
at getHooks.reduce.then (dev\serverless\lib\classes\PluginManager.js:540:24)
From previous event:
at PluginManager.run (dev\serverless\lib\classes\PluginManager.js:540:8)
at variables.populateService.then (dev\serverless\lib\Serverless.js:115:33)
at runCallback (timers.js:810:20)
at tryOnImmediate (timers.js:768:5)
at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
at Serverless.run (dev\serverless\lib\Serverless.js:102:74)
at serverless.init.then (dev\serverless\bin\serverless.js:72:30)
at dev\serverless\node_modules\graceful-fs\graceful-fs.js:111:16
at dev\serverless\node_modules\graceful-fs\graceful-fs.js:45:10
at FSReqWrap.oncomplete (fs.js:135:15)
From previous event:
at initializeErrorReporter.then (dev\serverless\bin\serverless.js:72:8)
at runCallback (timers.js:810:20)
at tryOnImmediate (timers.js:768:5)
at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
at Object.<anonymous> (dev\serverless\bin\serverless.js:57:4)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Function.Module.runMain (module.js:693:10)
at startup (bootstrap_node.js:191:16)
at bootstrap_node.js:612:3
The log mentioned at the start of the error report has already been deleted by the rollback before I can get to it.
From digging around the template file, it is clear that the execution of the custom resource javascript fails during the CREATE event for S3uploadedCustomS31. The function CustomDashresourceDashexistingDashs3LambdaFunction is the one that is executed in this case and it has been created as is seen in the log. That function does two things in succession
One of these is causing the failure but I have no way of determining which, partly because the error message is vague and partly because the entire stack gets rolled back, deleting the logs.
function create(event, context) {
const { Region, AccountId } = getEnvironment(context);
const { FunctionName, BucketName, BucketConfigs } = event.ResourceProperties;
const lambdaArn = getLambdaArn(Region, AccountId, FunctionName);
return addPermission({
functionName: FunctionName,
bucketName: BucketName,
region: Region,
}).then(() =>
updateConfiguration({
lambdaArn,
region: Region,
functionName: FunctionName,
bucketName: BucketName,
bucketConfigs: BucketConfigs,
})
);
}
Is there anything else I can do to get more info?
try to check: s3 -> bucket -> configuration -> events delete existing events it helps when you cannot deploy sls application
We are facing same error with existing:true
, seems like current workaround is only using manual trigger creation in AWS Lambda console
I get this when I set existing: true
to the s3 events trigger of a lambda function.
Are you running this with some limited rights on account, or maybe relying on cfnRole
setting?
It looks that custom resource was not able to apply needed configuration due to access issue.
In my case I had to change
events:
- s3:
existing: true
to
events:
- s3:
existing: true
(notice the two extra spaces)
It's weird how the error message is so cryptic though
@tommedema it's problem of lack of validation on Framework side. We plan to solve it with: https://github.com/serverless/serverless/issues/6562
So I've checked the spacing on my serverless.yml
. But even with existing: true
CloudFormation tries to create the S3 bucket and fails because it already exists.
events:
- s3:
existing: true
An error occurred: S3Bucket{name} already exists.
@jdelaune please share full content of serlveress.yml
best if it's minimal (no plugins involved), with sensitive parts masked out, and confirmed to expose the issue
service: img-resizer
frameworkVersion: ">=1.0.0 <2.0.0"
provider:
name: aws
stage: ${self:custom.ci.cf_environment}
runtime: nodejs12.x
region: ${self:custom.ci.aws_region}
deploymentBucket:
name: ${self:custom.ci.aws_bucket_deployment}
serverSideEncryption: AES256
iamRoleStatements:
- Effect: "Allow"
Action:
- "ec2:CreateNetworkInterface"
- "ec2:DescribeNetworkInterfaces"
- "ec2:DetachNetworkInterface"
- "ec2:DeleteNetworkInterface"
Resource: "*"
- Effect: "Allow"
Action:
- "s3:PutBucketNotification"
Resource:
Fn::Join:
- ""
- - "arn:aws:s3:::"
- ${self:custom.ci.aws_bucket_spot_photo_uploads}
- Effect: "Allow"
Action:
- "s3:GetObject"
- "s3:PutObject"
- "s3:PutObjectAcl"
Resource:
Fn::Join:
- ""
- - "arn:aws:s3:::"
- ${self:custom.ci.aws_bucket_spot_photo_uploads}
- "/*"
vpc:
securityGroupIds: ${self:custom.ci.aws_vpc_security_group_ids}
subnetIds: ${self:custom.ci.aws_vpc_subnet_ids}
custom:
ci: ${file(params.json):ci}
functions:
thumbnailise:
name: img-resizer-${self:custom.ci.cf_environment}
description: Creates thumbnails for new images on S3
handler: handler.thumbnailise
memorySize: 128
timeout: 30
events:
- s3:
existing: true
bucket: ${self:custom.ci.aws_bucket_spot_photo_uploads}
event: s3:ObjectCreated:*
rules:
- suffix: .o.jpg
@jdelaune thanks, can you provide also full output of sls deploy
as run with SLS_DEBUG=*
flag?
My apologies, figured out my issue was our CI server had an old version of serverless on it before the existing property was supported. It works as expected.
I am getting an error when existing:true
creating the internal Iam role, as all roles created in my aws account should have the permission boundary defined.
An error occurred: IamRoleCustomResourcesLambdaExecution - API: iam:CreateRole User: arn:aws:sts::xxx is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::yyy:role/xxx-IamRoleCustomResourcesLa-ZZZZ
Did anyone find a solution?
Updating to version 1.77.0 fixed it for me
If anybody faces the issue bellow AND is using custom cfn
role for deployment, please read solution here after.
Serverless Error ---------------------------------------
ServerlessError: An error occurred: S3uploadedCustomS31 - Failed to create resource. Access Denied See details in CloudWatch Log: 2019/10/01/[$LATEST]0d3cb59b1731410db457c06299c4df35.
Ensure your custom cfn role has the following actions allowed :
the following 4 actions related to lambda trigger configuration handled by the customResource lambda doing provisioning.
s3:PutBucketNotification
s3:GetBucketNotification
lambda:AddPermission
lambda:RemovePermission
the following 2 actions to ensure cfn can create a role for the lambda above and pass its allowed actions to the newly created role
iam:CreateRole
iam:PassRole
In my case I had to change
events: - s3: existing: true
to
events: - s3: existing: true
(notice the two extra spaces)
It's weird how the error message is so cryptic though
Yeah, this is also crazy, and still ongoing. They need to fix that delta in yaml config for s3. Totally maddening, imo
Yeah, this is also crazy, and still ongoing. They need to fix that delta in yaml config for s3. Totally maddening, imo
@wkhatch Such configuration errors are now neatly reported by our config schema based validation. Ensure to use latest version of a Framework
Hi All,
I have something like that error, my configuration serverless it's the following:
service: NAMESERVICE
provider:
name: aws
runtime: dotnetcore3.1
timeout: 10
autoPublishAlias: live
region: us-east-2 # AWS region
deploymentBucket:
name: BUCKETDEPLOY
iam:
role: NAMEROLE
deploymentRole: DEPLOYROLE
vpc:
securityGroupIds:
- SECURITYGROUPID
subnetIds:
- SUBNETIDID
package:
artifact: CACHEPACKAGE
functions:
eventRequest:
name: NAMELAMBDA
handler: HANDLERAPI
events:
- s3:
bucket: NAMES3
event: s3:ObjectCreated:*
rules:
- prefix: waves/
- suffix: .csv
existing: true
But at the time of deploying it gave me the following error:
Serverless Error ----------------------------------------
An error occurred: CustomDashresourceDashexistingDashs3LambdaFunction - Resource handler returned message: "The role defined for the function cannot be assumed by Lambda. (Service: Lambda, Status Code: 400, Request ID: f079df33-4ddf-4543-bb49-517b2fd43fc7, Extended Request ID: null)" (RequestToken: bff75f0e-1f65-795c-3b03-84f7192b2913, HandlerErrorCode: InvalidRequest).
Please, i need your help
@d3m0n1n it appears that role you list at iam.deploymentRole
doesn't have "lambda.amazonaws.com"
listed as one of the trusted principal services
Hi @medikoo,
But the truth is that the doploymentrole has the trust permissions
the problem is when I remove "existing: true" in serverless.yml, cloudformation try to create resource s3 that exists, right now and when I use "existing: true", cloudformation fails giving the above mentioned error
I read in the documentation that createRole
was required b/c I was trying to use an existing bucket, so, fine, rather than create new permission boundaries etc etc I'll just use a new bucket. Nope! Serverless really really really wants createRole
and I'd prefer to do almost anything to avoid giving it createRole
, since unfamiliar territory in AWS security is not somewhere I want to experiment in production. About to give up on Serverless entirely.
Output:
An error occurred: IamRoleLambdaExecution - API: iam:CreateRole User: arn:aws:sts::123:assumed-role/serverless-cloudformation/AWSCloudFormation is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::123:role/redacted-function-name-serverless-us-east-1-lambdaRole.
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information ---------------------------
Operating System: linux
Node Version: 14.18.1
Framework Version: 2.64.1
Plugin Version: 5.5.0
SDK Version: 4.3.0
Components Version: 3.17.1
Here's my config:
circleci@abcd:~/project/services/redacted$ cat serverless.yml
service:
name: redacted
frameworkVersion: "2"
provider:
name: aws
runtime: nodejs14.x
lambdaHashingVersion: 20201221
stage: serverless
region: us-east-1
deploymentBucket:
name: redacted-${self:service.name}
blockPublicAccess: true
serverSideEncryption: AES256
iam:
deploymentRole: arn:aws:iam::123:role/serverless-cloudformation
functions:
redacted:
handler: handler.run
name: serverless-${self:service.name}
events:
- s3:
bucket: serverless-redacted-bucket-name
event: s3:ObjectCreated:*
rules:
- prefix: redacted-prefix/
- suffix: redacted_suffix
plugins:
- serverless-deployment-bucket
@jazwiecki - The error you're seeing it unrelated to S3 bucket - if you don't specify a role for Lambda functions to use, one will be created for you by Serverless Framework and used by your functions. You can avoid that by specifying an existing role. You can read more about it here: https://www.serverless.com/framework/docs/providers/aws/guide/iam#iam
That did it! Thank you.
Glad to hear that @jazwiecki :raised_hands:
Hi @medikoo,
But the truth is that the doploymentrole has the trust permissions
the problem is when I remove "existing: true" in serverless.yml, cloudformation try to create resource s3 that exists, right now and when I use "existing: true", cloudformation fails giving the above mentioned error
Did you manage to solve this? I still have this issue
I have a very similar issue, the difference is that the deployment works but sls remove gives me this error:
CloudFormation - DELETE_FAILED - Custom::S3 - DataImportCustomS31
This bucket was not created by myself and is not specified in any serverless.yaml
I am alos using existing: true
on a s3 trigger
I also encountered this a while ago. It turns out that my bucket name is correct. The version that I'm using is:
$ serverless -v
Framework Core: 3.18.0
Plugin: 6.2.2
SDK: 4.3.2
TLDR; The fix is to update Serverless to the latest version and make sure that your S3 bucket name is correct.
If anybody faces the issue bellow AND is using custom
cfn
role for deployment, please read solution here after.Serverless Error --------------------------------------- ServerlessError: An error occurred: S3uploadedCustomS31 - Failed to create resource. Access Denied See details in CloudWatch Log: 2019/10/01/[$LATEST]0d3cb59b1731410db457c06299c4df35.
Ensure your custom cfn role has the following actions allowed :
the following 4 actions related to lambda trigger configuration handled by the customResource lambda doing provisioning.
s3:PutBucketNotification
s3:GetBucketNotification
lambda:AddPermission
lambda:RemovePermission
the following 2 actions to ensure cfn can create a role for the lambda above and pass its allowed actions to the newly created role
iam:CreateRole
iam:PassRole
So would all 6 of these abilities need to be part of the role referenced in 'role: role-arn-here' or the 'deploymentRole: role-arn-here' ?
We ran into this issue. Our deployment role does not have iam:CreateRole
so if we try to create triggers on an existing bucket we get an error when SLS tries to create IamRoleCustomResourcesLambdaExecution
.
It looks like this role is created as the lambda execution role for a custom lambda that adds the event trigger to S3. It is always created if you're using an existing bucket. See here and here.
There's an analogous feature in the console where it grant S3 the permissions to invoke the bucket.
We solved this problem by manually adding the event triggers after deploying the lambda.
If you are looking for future features it would be nice if there was a way to specify the execution role for the custom lambda.
If you are looking for future features it would be nice if there was a way to specify the execution role for the custom lambda.
@sblack4 custom lambda covers deployment steps that cannot be done via CF, so technically it should work with the same deployment role used for CF deployments.
Also, ideally, if there would be no custom resource involved, and in some cases, we managed to get rid of that requirement
If you are looking for future features it would be nice if there was a way to specify the execution role for the custom lambda.
@sblack4 custom lambda covers deployment steps that cannot be done via CF, so technically it should work with the same deployment role used for CF deployments.
Also, ideally, if there would be no custom resource involved, and in some cases, we managed to get rid of that requirement
You're right on both points. Users could use the same role for the custom lambda as they do for the main lambda. Rather than give them a boolean to do just that I think it would be simpler to optionally specify the role by name/arn. Then if they want to create separate roles they can
Hi @medikoo ,
I'm struggling with the same issue where the CF fails after some time with:
CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [1d625252-bc94-4e3a-8146-5dfc500f2707]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
The thing is, that I know the feature of linking events to an existing s3 works as I've been doing it on another project. However this project is a bit different in setup on AWS level.
I was able to get some logs from the custom resource errors. Hopefully this can shed some lights to someone here and lead us to the right direction:
{
"time": "2024-01-29T18:32:22.371Z",
"type": "platform.initReport",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.ExitError",
"metrics": {
"durationMs": 162.964
}
}
}
{
"time": "2024-01-29T18:32:22.371Z",
"type": "platform.initRuntimeDone",
"record": {
"initializationType": "on-demand",
"phase": "init",
"status": "error",
"errorType": "Runtime.ExitError"
}
}
{
"timestamp": "2024-01-29T18:32:22.351Z",
"level": "ERROR",
"message": {
"errorType": "ImportModuleError",
"errorMessage": "Error: Cannot find module 'aws-sdk'\nRequire stack:\n- /var/task/utils.js\n- /var/task/s3/lib/permissions.js\n- /var/task/s3/handler.js\n- /var/runtime/index.mjs",
"stackTrace": [
"Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'",
"Require stack:",
"- /var/task/utils.js",
"- /var/task/s3/lib/permissions.js",
"- /var/task/s3/handler.js",
"- /var/runtime/index.mjs",
" at _loadUserApp (file:///var/runtime/index.mjs:1087:17)",
" at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)",
" at async start (file:///var/runtime/index.mjs:1282:23)",
" at async file:///var/runtime/index.mjs:1288:1"
]
}
}
The serverless config is a bit hard to copy paste, but it is very basic approach:
someFunc:
handler: src/functions/some/handler.main
environment:
...
events:
- s3:
event: s3:ObjectCreated:*
bucket: some-bucket
existing: true
rules:
- prefix: some/prefix
If you need any other info let me know.
Cheers
@tonivdv This error signals that you've moved to the Node.js runtime which doesn't host aws-sdk
(AWS SDK v2) anymore.
If require of aws-sdk
is implied by the Framework custom resource logic, then ensure to rely on latest version of the Framework. If it happens that Framework is not updated yet, find the related issue here and check the progress (note I'm no longer with the Serverless Inc, and I'm unable to push any work forward here).
Side solution could be to add aws-sdk
to your service dependencies, and that should make your lambda running
Hey @medikoo ,
Thanks for reacting so fast despite you're not part of Serverless Inc anymore.
The runtime it is running in is fine:
But I'm wondering ... the major difference with the other project is that this serverless project runs lambdas inside the vpc ... So could it be that this internal custom resource runs lambda's in the normal way and can't access the lambda inside the vpc to do the configuration?
@tonivdv this runtime definitely doesn't host aws-sdk
, and error is Error: Cannot find module 'aws-sdk'
@medikoo but are we influencing anything on how we use serverless that can cause this? Because that custom resource stuff is internally generated by serverless framework right?
Ok, so I think I understand ... the custom resource generated by Serverless framework is generating this with aws sdk 2 which is not available since node 18 ... and this is causing the issue ... If my understanding is correct, can I fix that easily or is there a internal fix needed inside serverless framework?
Hm I guess this is in relation with https://github.com/serverless/serverless/issues/12133 and PR https://github.com/serverless/serverless/pull/12146
I am getting this error while trying to deploy lambda using serverless. 19:05:38 Error: 19:05:38 CREATE_FAILED: ConvertDashtoDashscoreCustomS31 (Custom::S3) 19:05:38 Received response status [FAILED] from custom resource. Message returned: Access Denied See details in CloudWatch Log: 2024/03/26/[$LATEST]cbbc894c40b74bc8b105a8905c403b74 (RequestId: bd7aa89f-2cca-4965-9374-95a14d1b43e5)
@tonivdv were you able to figure out how to package the aws-sdk dependency with the custom resource function? I'm not seeing a clear path here.
@medikoo could you possibly expand on this:
Side solution could be to add aws-sdk to your service dependencies
Specifically, how would I add aws-sdk
to the dependencies for the custom resource functions (e.g. myservice-release-custom-resource-existing-s3
)?
@cwinters8 unfortunately not :(
Specifically, how would I add aws-sdk to the dependencies for the custom resource functions (e.g. myservice-release-custom-resource-existing-s3)?
@cwinters8 yes, that I think was mistake on my side. I overlooked that framework packages custom resource lambdas independently and what dependencies you define in service is not relevant.
I assume as long Framework is not upgraded the only solution is to hack it somehow (via plugin). I take it'll be possible but will require some understanding of its internals.
Bug Report
Setting existing:true on S3 bucket causes deployment to fail
Description
What did you do? Added existing:true to S3 function. The deployment works correctly for a new bucket.
What happened? The deployment fails with:
and then rolls back everything else. The rollback for S3uploadedCustomS31 fails of course.
What should've happened? Deployment should complete. The step that fails is the custom resource handler that attaches the necessary policies to the function handler and the existing bucket. It is not clear which step fails. I suspect it is attaching the permission to the lambda.
What's the content of your
serverless.yml
file? Reduced file for clarityprovider: name: aws runtime: nodejs10.x stage: ${opt:stage, 'dev'} region: ${opt:region, 'sa-east-1'}
iamRoleStatements:
functions: s3uploaded: handler: src/uploaded.handler events:
SLS_DEBUG=*
environment variable (e.g.SLS_DEBUG=* serverless deploy
) sls does not fail, but the AWS deployment does.Similar or dependent issues:
6638 This gave a different error.
Because the whole stack rolls back due to the error, there are no AWS logs to track down the issue. Debugging CloudFormation custom resources is a bit of a dark art.