No longer can register 'Community::Organizations::NoDefaultVPC' resource

[sshvetsov]

Yesterday (2023-05-30) I tried to register Community::Organizations::NoDefaultVPC in additional AWS accounts in my organization and the register-type task failed in every region of the 17 default AWS regions and across several different AWS accounts with error messages like these:

INFO: Type registration Community::Organizations::NoDefaultVPC is in DEPLOY_STAGE of status FAILED. waiting....
ERROR: Workload NoDefaultVpcType in 111111111111/us-west-1 update failed. reason: Registration of Resource Type Community::Organizations::NoDefaultVPC failed. Deployment is currently in DEPLOY_STAGE of status FAILED
Deployment failed with error. Error message: [3e06a148-45a3-4ee5-9455-f3eb9a89458a] Internal Failure (111111111111 = SomeAccount)

I've tried registering the type manually using the steps documented here but with the same error: Deployment is currently in DEPLOY_STAGE of status FAILED\nDeployment failed with error. Error message: [9726a281-f642-4a09-8c0c-07eee753d9b6] Internal Failure.

I've successfully registered this custom type in a handful of accounts before using OFN and the manual steps, so it seems like something may have changed on the AWS side.

Here's the redacted output of the step-by-step manual registration process for reference:

1. first install the execution role

   aws cloudformation create-stack \
   --template-url https://community-resource-provider-catalog.s3.amazonaws.com/community-organizations-nodefaultvpc-resource-role-0.1.0.yml \
   --stack-name community-organizations-nodefaultvpc-resource-role \
   --capabilities CAPABILITY_IAM
   {
   "StackId": "arn:aws:cloudformation:us-east-1:111111111111:stack/community-organizations-nodefaultvpc-resource-role/f9749710-ff69-11ed-b446-123ab557fdf9"
   }

2. get the value of the ExecutionRoleArn Output

   aws cloudformation describe-stacks \
   --stack-name community-organizations-nodefaultvpc-resource-role
   {
   "Stacks": [
       {
           "StackId": "arn:aws:cloudformation:us-east-1:111111111111:stack/community-organizations-nodefaultvpc-resource-role/f9749710-ff69-11ed-b446-123ab557fdf9",
           "StackName": "community-organizations-nodefaultvpc-resource-role",
           "Description": "This CloudFormation template creates a role assumed by CloudFormation during CRUDL operations to mutate resources on behalf of the customer.\n",
           "CreationTime": "2023-05-31T04:16:54.272000+00:00",
           "RollbackConfiguration": {},
           "StackStatus": "CREATE_COMPLETE",
           "DisableRollback": false,
           "NotificationARNs": [],
           "Capabilities": [
               "CAPABILITY_IAM"
           ],
           "Outputs": [
               {
                   "OutputKey": "ExecutionRoleArn",
                   "OutputValue": "arn:aws:iam::111111111111:role/community-organizations-nodefaultvpc-ExecutionRole-EG67DEIOE220"
               }
           ],
           "Tags": [],
           "EnableTerminationProtection": false,
           "DriftInformation": {
               "StackDriftStatus": "NOT_CHECKED"
           }
       }
   ]
   }

3. register the cloudformation type

   aws cloudformation register-type \
   --type-name Community::Organizations::NoDefaultVPC \
   --type RESOURCE \
   --schema-handler-package s3://community-resource-provider-catalog/community-organizations-nodefaultvpc-0.1.0.zip \
   --execution-role arn:aws:iam::111111111111:role/community-organizations-nodefaultvpc-ExecutionRole-EG67DEIOE220
   {
   "RegistrationToken": "06381ea4-0c68-49de-a15c-fd3f7254c190"
   }

4. describe type registration

   aws cloudformation describe-type-registration --registration-token 06381ea4-0c68-49de-a15c-fd3f7254c190
   {
   "ProgressStatus": "FAILED",
   "Description": "Deployment is currently in DEPLOY_STAGE of status FAILED\nDeployment failed with error. Error message: [06381ea4-0c68-49de-a15c-fd3f7254c190] Internal Failure",
   "TypeArn": "arn:aws:cloudformation:us-east-1:111111111111:type/resource/Community-Organizations-NoDefaultVPC",
   "TypeVersionArn": "arn:aws:cloudformation:us-east-1:111111111111:type/resource/Community-Organizations-NoDefaultVPC/00000001"
   }

Any idea what could be causing this, or how I can troubleshoot this further?

[yannickvr]

+1, I was/am experiencing the same, but with the S3PublicAccessBlock.

I was unable to remove the type registration, as the registration does not have a type (at least, that was the error the CLI gave)

[OlafConijn]

I have seen this on occasion too. Typically multiple people using orgformation report this and (to date) a day or so later something seems to get fixed at AWS.

What I think would help is open a support case

[sshvetsov]

I will try opening a support case w/ AWS on this. UPDATE: Support case opened.

[sshvetsov]

@OlafConijn, crazy thought: could the version of the nodejs used in the custom resources be what's causing the issue? The .rpdk-config has runtime nodejs12.x. If this is the same as Lambda runtimes, the nodejs12.x entered the "Phase 1" of the deprecation on Mar 31, 2023.

From https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Deprecation (end of support) for a runtime occurs in two phases.

Phase 1 - Lambda no longer applies security patches or other updates to the runtime. You can no longer create functions that use the runtime, but you can continue to update existing functions. This includes updating the runtime, and rolling back to the previous runtime. Note that functions that use a deprecated runtime are no longer eligible for technical support.

This would explain why we can't register a resource type if it's equivalent to creating a new Lambda function with deprecated runtime.

[sshvetsov]

Thanks, @OlafConijn, for repackaging the 0.1.0 version of the Community::Organizations::NoDefaultVPC resource type with updated runtime. This solves the problem for this particular resource type, but now that we've confirmed that the deprecated Node.js runtime version was the cause of the problem, all other resource types in this repo still using the nodejs12.x runtime need to be updated as well.

[sshvetsov]

Side note, AWS support did a great job figuring this issue out. They got back to me within an hour after Olaf fixed this issue with the same conclusion and recommendation.

[OlafConijn]

thanks so much for your help on this @sshvetsov. I just finished updating the resource providers in source + registry.