Error executing "PublishVersion" with StateReasonCode: 'Creating'

[jmariklecr]

Hello. We're getting the following error:

Deploying viewer request Lambda function
Error: Error executing "PublishVersion" on "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###################################-Viewer-Request/versions"; AWS HTTP error: Client error: `POST https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###################################-Viewer-Request/versions` resulted in a `409 Conflict` response:
{"Type":"User","message":"The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:########### (truncated...)
 ResourceConflictException (client): The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:###########:function:###################################-Viewer-Request is currently in the following state: 'Pending'. StateReasonCode: 'Creating' - {"Type":"User","message":"The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:###########:function:###################################-Viewer-Request is currently in the following state: 'Pending'. StateReasonCode: 'Creating'"}

The issue appears to be tied to a logic error in the following section:

https://github.com/dyerc/craft-flux/blob/c5e5c39f6729ec3737e8002bbfa4b0f494718b8c/src/services/Lambda.php#L304-L314

This area of the code is looking for LastUpdateStatus, but getFunction doesn't return LastUpdateStatus on the root level of the response array. It's nested in the Configuration array. I believe the Flux plugin should either be checking $response['Configuration']['LastUpdateStatus'] or it should just be calling getFunctionConfiguration instead of getFunction in this section. This would match the shape of what the existing code is expecting.

The result is that the mentioned error occurs. Lambda is still in the process of creating the function when a version push is called. It's supposed to check to see if the function is ready to be changed, but it's failing to correctly reference where this is indicated in the API response. This is preventing the deployment of the lambda functions that Flux relies on for our projects.

A fix would be much appreciated. Thank you.

[dyerc]

Thanks very much @jmariklecr for investigating this further. Yes, I almost wonder how it's gone so long without noticing this issue before, because this code isn't correctly checking the LastUpdateStatus!

I have released a fix for the v5 branch. Would you be able to confirm this sorts the issue for you please? If so, I will backport this to the v4 branch too.

[jmariklecr]

Thanks for looking into it, @dyerc. I agree about it being odd that it worked up till this point. My guess is that AWS wasn't enforcing the block to changes while creating until recently. Maybe it was only a warning before, or maybe they queued the request. Still, I'm glad it's a simple fix.

We'll have an opportunity to test this next week. I'll follow up after we do.

[strandofgenius]

@dyerc with your fix it does get past the initial issue. I believe Flux needs to wait after calling publishVersion at line 192, otherwise the LastUpdateStatus will report as InProgress the first time that the install script runs. It appears that subsequent installation attempts work because the publishVersion has finished.

https://github.com/dyerc/craft-flux/blob/c5e5c39f6729ec3737e8002bbfa4b0f494718b8c/src/services/Lambda.php#L192

[bfopma]

@dyerc following up on this one -- we have a Craft 4 site scheduled to launch tomorrow and the current Flux issue is problematic. We can likely launch without it but would like to get this resolved as soon as possible.

[dyerc]

Sorry this took me a while to get a fix published @strandofgenius and @bfopma , I've been away for the last week. Hopefully the above commit and version 5.0.2 will add a suitable wait. Please let me know if this resolves the issue for you?

[jmariklecr]

@dyerc Back with another test. Sorry these take a long time between each other. With it being an initialization step, we don't have to do it often. Unfortunately, we did still run into an issue with the latest version (5.0.3). Here's the output:

$ php craft flux/aws/install Deploying viewer request Lambda function Deploying origin response Lambda function Error: Error executing "PublishVersion" on "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/######################-Origin-Response/versions"; AWS HTTP error: Client error: POST https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/######################-Origin-Response/versions resulted in a 409 Conflict response: {"Type":"User","message":"The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:######## (truncated...) ResourceConflictException (client): The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:############:function:######################-Origin-Response is currently in the following state: 'Pending'. StateReasonCode: 'Creating' - {"Type":"User","message":"The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:############:function:######################-Origin-Response is currently in the following state: 'Pending'. StateReasonCode: 'Creating'"}

It looks like it's still running into an unhandled exception. Based on the output, I would assume it's happening during the origin response lambda deploy. Running the install command a second time runs without issue. Unfortunately, I am tight on time due to a project being delayed partway into when it was supposed to start, so I can't provide any thorough investigation for this one. If I find more time later, I'll try to figure out a way to reproduce this more consistently without having to wait for the start of an entire project.