Open jandppw opened 3 years ago
In a further attempt to get this working again, I added --aws-delay 10000 ---aws-retry 30
to the update command.
No joy. Fails just as fast.
Monitoring with
> watch -n1 aws --profile YYYYYYY --region eu-west-1 lambda get-function-configuration --function-name XXXXXXXX
this time showed no other LastUpdateStatus
than "Successful"
. It might have happened in between polls.
LastModified
and RevisionId
did change once, but the CodeSha256
and CodeSize
did not.
So something (the configuration?) did change, but the code did not.
Created branches, which only deploy in CI, for the 2 last successful deploys, which worked a gazillion times before.
Both experiments get the same problem now.
Since package-lock.json
was not changed, and deploy happens with npm ci
, this can only mean something at the AWS side changed, either in general, or in this particular instance.
We just ran into the same issue and found the reason here:
https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
FYI: We upgraded the version of the aws provider in one of our terraform sets to the latest version and that seems to have cleared the problem.
@choeller thx for the response
That is consistent with our observations.
We just ran into the same issue and found the reason here:
https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
FYI: We upgraded the version of the aws provider in one of our terraform sets to the latest version and that seems to have cleared the problem.
@spencer-aerian that cleared the problem in claudia
?!?
Or are you suggesting updating the version of the AWS SDK used inside claudia
?
Dear Claudiajs,
Thx for creating and maintaining this project. We've been using it for deployment for a number of years now (and not other features).
This issue is blocking for us. There has been little activity here over the last few months. Can you give us an indication about your intentions with this project?
I would like to be able to determine whether it is worth waiting for further progress, or whether it is more appropriate to look for another long term solution.
FYI: We upgraded the version of the aws provider in one of our terraform sets to the latest version and that seems to have cleared the problem.
@spencer-aerian that cleared the problem in
claudia
?!?Or are you suggesting updating the version of the AWS SDK used inside
claudia
?
No different package but same error.
I just checked out the project, and ran the tests (this takes ages).
There are some errors.
Notably, there are frequent mentions of
error cleaning up TooManyRequestsException: Too Many Requests
so I guess these tests are leaving behind some flutsam in our AWS account now.
Also, there are failures with update
/ environment variables
. As far as I can see, there are more keys in the object returned than expected. AWS_LAMBDA_INITIALIZATION_TYPE
seems unexpected.
But the heart of this issue is The operation cannot be performed at this time. An update is in progress …
. And that message is exactly what we get with tests that work with layers. All those tests fail with this message.
update
/ layer support
all fail with this message.
I am running the tests with both the AWS-SDK defined in the project, and the most recent version, and they fail with the latest AWS-SDK. In other words, updating the AWS SDK is not the answer in this case (it was fairly recent to start with).
@jandppw This pr fixes it for me: https://github.com/zappa/Zappa/pull/992
Getting this problem here, unable to deploy an update of my lambda in production
v5.14.0 should fix this
@gojko - Still getting this on v5.14.0
@gojko - Also, still getting this on v5.14.0 I am using api-gateway and have a temporary work around by creating the gateway under a different name (that works just fine). This just started happening for me today - I successfully updated yesterday. Also I just upgraded to latest aws-sdk v2.1031.0
Maybe irrelevant and not applicable, but I solved this by adding the env var to my Cloudformation template instead of during deploy.
Unrelated: I don't use this library, but looks like the code now has to waitFor()
the previous function update before running further updates.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html
For example, first the function is created and then its configuration is updated — has to wait in-between. Same is for updating the function's *.zip file and then reconfiguring it — has to wait too.
log('Updating lambda function');
// Update lambda code
await lambdaApi.updateFunctionCode({
FunctionName: getFunctionName(lambdaConfig, stage),
ZipFile: zipFile
}).promise();
log('Updated code');
// Has to wait for the function update status to transition to "Updated"
// until making further re-configuration.
// https://github.com/claudiajs/claudia/issues/226
// https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
// https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html
await lambdaApi.waitFor('functionUpdated', {
FunctionName: getFunctionName(lambdaConfig, stage)
}).promise();
// Update lambda configuration
await lambdaApi.updateFunctionConfiguration(lambdaConfiguration).promise();
log('Updated configuration');
Also just started happening to me today, both 5.14.0 and 5.13.0
Started happening to me today as well. updating to the latest ClaudiaJs and AWS-SDK didn't help.
I was however able to mitigate the error by adding the aws:states:opt-out
in the description of the lambda function or by not passing the environment variables during deploy.
Started for us as well. if someone wants to do quick fix until its fixed in ClaudiaJs, use below around Claudia commands:
aws lambda update-function-configuration --function-name ${FuncName} --description "aws:states:opt-out"
claudia update
aws lambda update-function-configuration --function-name ${FuncName} --description ""
Upgrade zappa to latest version. Worked for me.
I started making skills for Alexa for about a month. I was doing that using ASK CLI. This morning this error popped out:
[Error]: CliError: The lambda deploy failed for Alexa region "default": ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:454044463163:function:ask-reading-buses-default-default-1636455775666
Now I can't deploy any skill.
I all of a sudden had the same problem using the same program I have always used to deploy. I was updating the lambda code followed by a publish version. I would get the error "An update is in progress for resource". I fixed it by calling the getFunctionConfiguration and waiting til State was Active and LastUpdateStatus was Successful. I then continued on with the publish. The State was Active right away, but now it took the LastUpdateStatus to be Successful about 30 seconds or so.
I was in trouble with the same error on Node 12. Just update the Node version to 14 and it's working for me.
It looks like this is only still an issue if updateConfiguration is called with any options.
For me, removing the --runtime
argument allowed the function to deploy correctly.
@gojko Something similar to the following should be enough to fix this (just copied the existing wait logic up into the updateConfiguration function)
--- a/src/commands/update.js
+++ b/src/commands/update.js
@@ -145,7 +145,11 @@ module.exports = function update(options, optionalLogger) {
},
() => logger.logStage('waiting for IAM role propagation'),
Promise
- );
+ ).then(result => {
+ logger.logStage('waiting for lambda resource allocation');
+ return waitUntilNotPending(lambda, lambdaConfig.name, awsDelay, awsRetries)
+ .then(() => result);
+ });
}
},
cleanup = function () {
This is still broken when using --set-env-from-json
This is still broken for me as well. I temporarily bypassed the situation by adding the opt out clause (aws:states:opt-out) to the Lambda description field, but this trick will only work until December 6th.
@magoyo Same here, However, i using code-build which is similar. So, the solution is add --description "aws:states:opt-out"
for any update?
opt-out
temporary plan
@ambigus9 I didn't see a --description
option in the update
documentation (it was only on create
), so I added it to the project description in package.json and also to the description field in Lambda configuration screen. It worked like a charm, but it's obviously temporary and we don't have much time.
I was able to fix this locally as follows:
src/commands/update.js
file.src/tasks/wait-until-not-pending.js
file to include a new line:await new Promise(resolve => setTimeout(resolve, timeout));
And making the waitUntilNotPending async
.
So now it looks like this:
const retry = require("oh-no-i-insist");
module.exports = async function waitUntilNotPending(lambda, functionName, timeout, retries) {
'use strict';
await new Promise(resolve => setTimeout(resolve, timeout));
return retry(
() => {
return lambda.getFunctionConfiguration({FunctionName: functionName}).promise()
.then(result => {
if (result.state === 'Failed') {
throw `Lambda resource update failed`;
}
if (result.state === 'Pending') {
throw 'Pending';
}
});
},
timeout,
retries,
failure => failure === 'Pending',
() => console.log('Lambda function is in Pending state, waiting...'),
Promise
);
};
I am not sure if this is the correct approach, but it works without the need to update the description for now.
@magoyo Ok, but i using codebuild on AWS, so for the solution was add sleep 90
which in linux instance means a delay of 90 seconds, resulting on a temporal solution.
@ambigus9 I don't use codebuild, but I understand your approach.
@maltahs Thank you for expanding on @hexid suggestion. I will be implementing your fix in my code unless this issue is resolved in the next couple of days.
@maltahs Your suggestion worked for us too, thanks! To make things easier for others, I also submitted our hotfix branch as a PR to this project (as you can see above).
@gojko, given the feedback above, shouldn't this issue be re-opened?
Are there any updates on a new release that would fix this issue (perhaps accepting PR #230?)?
+1 for accepting #230. Resolved the issue for me as well. Thanks @madve2 !!
+1 for accepting #230
+1 for accepting #230
We're considering other alternatives at the moment to claudia and we hope this is merged soon! Thanks @madve2
I'm trying to upload my Node function to Lambda with the vanilla aws-sdk package, and I'm getting the same error for my resource. It must be something to do with my Lambda configuration itself, not Claudia, but I have no idea how to fix it.
const aws = require('aws-sdk');
const lambda = new aws.Lambda({
region: 'us-east-1'
});
var params = {
FunctionName: functionName,
ZipFile: zipBuffer
};
console.log('Uploading function code...');
lambda.updateFunctionCode(params, (err, data) => {
if (err) cb(err);
else {
console.log(`Uploaded function code:\n\t FunctionName=${data.FunctionName}\n\t Role=${data.Role}\n\t CodeSha256=${data.CodeSha256}`);
console.log('Publishing new version...');
var params = {
CodeSha256: data.CodeSha256,
Description: `${time}`,
FunctionName: functionName,
};
lambda.publishVersion(params, (err, data) => {
if (err) cb(err);
else {
// continue
Output:
Uploading function code...
Uploaded function code:
FunctionName=ScoutradiozPrimaryStack-PrimaryFunction-1N6C440CXO15P
Role=arn:aws:iam::243452333432:role/ScoutradiozPrimaryStack-LambdaExecutionRole-RAGPEHPPHKZ3
CodeSha256=r6GcoXGmMp4Zg4V6MqDE/02X5T9PSQ4bTn3oe8VMgHQ=
Publishing new version...
/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/request.js:31
throw err;
^
ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:243452333432:function:ScoutradiozPrimaryStack-PrimaryFunction-1N6C440CXO15P
at Object.extractError (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/protocol/json.js:51:27)
at Request.extractError (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
at Request.callListeners (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/media/drak/DATA/OneDrive/Projects/Programming/ScoringApp-Serverless/node_modules/aws-sdk/lib/request.js:685:12) {
code: 'ResourceConflictException',
time: 2022-01-15T18:10:12.031Z,
requestId: 'f42cf16f-ca70-41a8-95a6-55a9751717e5',
statusCode: 409,
retryable: false,
retryDelay: 0.2627167491526139
}
Any idea what to do to fix it?
My bad! I didn't fully read through the comments. After adding @maltahs' waitUntilNotPending function to my own script, it works. I'm gonna keep my comment here just in case someone else is confused in a similar vein to how I was. I misunderstood this: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html thinking that the "update in progress" was something major on the back-end. But no, it was just the update that I had started just a few ms before I attempted to publish a new version.
This needs to be fixed otherwise we need to think about not using claudiajs anymore
I had to find another solution unfortunately now that its been months waiting on the pr
I had to find another solution unfortunately now that its been months waiting on the pr
@cathalelliott1 It's actually not too difficult to manage your Lambda functions manually with the aws-sdk package. If you need a reference, feel free to take a look at the scripts I made for my own project: https://github.com/FIRSTTeam102/ScoringApp-Serverless/tree/master/scripts - the scripts are called with NPM scripts as defined in https://github.com/FIRSTTeam102/ScoringApp-Serverless/blob/master/primary/package.json. Code is licensed with GPLv3 so feel free to use it if it helps. It's highly specific to the one project, but you can use the same concepts to fit your own use.
The scripts don't have many comments, but if you are looking into it and want an explainer on how they work, open an issue on our repo and I can answer any questions.
We are facing this issue as well and the code changes are small, so we were able to implement them locally (although it's a pain when you switch workstations and have to update again.)
So question - looking at the version history, it seems like after 2019 there have only been three releases (early 2020, early 2021, and October 2021). And there has been no activity here by the maintainers since the Oct 18 changes. Is ClaudiaJS no longer maintained, or is it just sporadically maintained now?
If ClaudiaJS is no longer maintained, then it would be good to know what packages are the easiest to transition to. Anyone has any good suggestions?
@magoyo Serverless Framework is the only option of which I'm aware. I'm sure there are others, I'm just not up to speed.
For those mentioning this is still open, did you try updating your package to 5.14.1? Looks like @gojko put in a commit that contains the suggestions by @hexid above https://github.com/claudiajs/claudia/commit/6284972467125b8e325899f3c0f740c2c03cd16b
Thank you @dmackinn, I hadn't used that version yet. It looks like he just patched it 21 days ago (after our commentary) in March 2022, not March 2021 like the date in the README.md says (and indeed there is a follow up commit where the year was changed from 2021 to 2022.) I'll give it a try.
I believe this was caused by AWS sending back State
with a capital "S" but Claudia checks for a lower-case state
so it doesn't wait for the state to change correctly. I encountered this running tests for #239, and included a fix in that PR.
In our project, Lambda was last deployed successfully by CI with
claudia
2021-09-14 ~16:17 CET. There have been no issues earlier.A next attempt by CI, at 2021-09-15 ~1649 CET failed. Retry attempts failed. Manual attempts via CLI failed. Manual upload, publish, and creation of an alias, worked via the console (but no working version was produced because there was no investment in getting the package right).
Nothing of relevance was changed (always a strong statement, I know). There was no update of
claudia
, or related packages between 2 deploys. A retry of the successful deployed version failed too.Retries 2021-09-16 ~ 10:10 CET failed again.
The reported error is always the same:
But this happens quick, before the package is build, or after.
We've felt for a while that
claudia
does some things twice, first checking, and then doing. When the error appears late, we see several mentions oflambda.setupRequestListeners
Resources on the internet are barely any help.
AWS Lambda - Troubleshoot invocation issues in Lambda mentions
ResourceConflictException
, but with a different message, and refers to VPCs, which we are not using.UpdateFunctionConfiguration
, PublishVersion,UpdateFunctionCode
and others mention more generally:Other resources are no help:
Terraform Error publishing version when lambda using container updates code #17153 (Jan. 2021) mentions a "lock" / "last update status", which we can watch during execution using
The output looks like
most of the time, but we see
LastUpdateStatus
change for a moment before the error occurs.Terraform aws_lambda_function ResourceConflictException due to a concurrent update operation #5154 says, in 2018,
serverless 'Concurrent update operation' error for multi-function service results in both deployment and rollback failure. #4964 reports the same issue in 2018, and remarks:
So, this appears to be a timing issue. Claudia should take it slower?