[BUG][Functional] Perimeter ALB Forwarding Lambda Function Error: Runtime.Unknown

[Ray-PHSA]

Required Basic Info

• Accelerator Version: 1.5.8-d
• Install Type: Upgrade
• Upgrade from version: 1.5.7-b

Describe the bug The perimeter ALB function that constantly checks the internal ALB targets for IP address changes started failing after upgrading to 1.5.8-d. The issue is related to the Lambda runtime upgrade to Nodejs 18.

The function name is: ASEA-Perimeter-Phase1-Vpc-PerimeteralbIpForwarding-tBMt3ocTnJHS

Failure Info

• What error messages have you identified, if any: The Lambda Cloudwatch logs show: INIT_REPORT Init Duration: 60257.48 ms Phase: invoke Status: error Error Type: Runtime.Unknown

• What symptoms have you identified, if any: The function fails 100% of the time and the IP addresses are not being updated in the external ALB targets which causes systems in Prod to go down!

Required files

• Please provide a copy of your config.json file (sanitize if required)

Steps To Reproduce

1. Make any change to the external ALB rules in the DynamoDB table
2. See the errors in the Lambda logs.

Expected behavior The Lambda should work normally and it should update the ALB targets (with the correct IP addresses) whenever they change.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

[Ray-PHSA]

Hi Brian, we fixed the Prod issue by increasing the Lambda memory to 512 MB. Please consider implementing that via code. Thanks!

[ebellavance]

Thank's Ray to report the problem. We just updated to 1.5.8-d and got the problem. We also increased the lambda memory to 512mb.

Do you know if running the statemachine (like creating a new account) will overide the change?

Regards,

Eric

[Ray-PHSA]

Hi, I'm not sure, I haven't run it since fixing the Lambda memory manually. I assume (based on the nature of CFN) that if the CFN synthesized template is not changing, CFN will not detect a drift and will leave the manual fix alone. Once you release 1.5.8-e (with the memory increase) then CFN will see a change and will attempt to update the function, then the function will be 'updated' from 512 (already done manually) to 512 (from the code) and the drift will be gone. BTW, how's the LZA upgrade script going? Thanks!

[ebellavance]

Ho, i'm not involved with ASEA or LZA development, i'm a customer using ASEA for his landing zone :)

[johnathan-tracz]

We are noticing that newly created forwarding rules do not seem to work after this update. We have increased the memory on the Lambda function to 512MB which got rid of the errors, but if we create a new forwarder, we see the following behaviour:

• Rule is successfully created on the ELB.
• Target group is successfully created.
• Target group does NOT register any targets - we have waited up to a couple hours with no change.

When checking the table entry in DynamoDB, the targetGroupIpAddresses entry never gets updated with any addresses.

Is anyone else seeing this behaviour?

[archikierstead]

Fixed in 1.5.9