Idempotency Gap # 1: CloudFormation auto-generation of values for `primary identifier` attributes when unspecified in the manifest

Overview

Every AWS resource type has a property that is defined as its primary identifier. The value of this property must be unique for each resource of that type in a given AWS account and AWS Region. For example, many resource types include a Name property that must be unique for each resource of that type. In some cases, the primary identifier is defined as a combination of multiple properties that together form a unique identifier. By using this primary identifier, combined with the resource type, you can specify exactly which resource on which you want to perform resource operations such as update-resource or delete-resource.

In addition, some resource types define secondary identifiers that can also be used to uniquely identify resources of that type.

To determine which resource property (or combination of properties) is the primary identifier for a resource type, refer to the primaryIdentifier attribute of the resource type schema. When the resource type includes secondary identifiers, refer to the additionalIdentifiers attribute of the resource type schema.

During the exploration to model AWS Resources with Bicep using the AWS Bicep Extensibility Provider types, we noticed that often the primaryIdentifier attributes of a resource are not required to be specified during resource creation as illustrated by Example 1.

Example 1

Consider a AWS::Kinesis::Stream resource, below are the relevant schema properties (see here for the full schema description)

{
  "arn": "arn:aws:cloudformation:us-west-2::type/resource/AWS-Kinesis-Stream",
  "description": "Resource Type definition for AWS::Kinesis::Stream",
  "type": "AWS::Kinesis::Stream",
  "schema": {
    "createOnlyProperties": [
      "/properties/Name"
    ],
    "primaryIdentifier": [
      "/properties/Name"
    ],
    "properties": {
      "Arn": {
        "description": "The Amazon resource name (ARN) of the Kinesis stream",
        "type": "string"
      },
      "Name": {
        "description": "The name of the Kinesis stream.",
        "maxLength": 128,
        "minLength": 1,
        "pattern": "^[a-zA-Z0-9_.-]+$",
        "type": "string"
      },
   //... Other Properties ... //
    },
    "readOnlyProperties": [
      "/properties/Arn"
    ],
    "sourceUrl": "https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-kinesis.git",
    "typeName": "AWS::Kinesis::Stream"
  }
}

From the JSON schema it follows that Name belongs to both createOnlyProperties and primaryIdentifier.

Notice that this schema in particular doesn't define the required attribute so the Name property isn't required to be specified for resource creation.

Further inspection of the resource type Name property in CloudFormation documentation reveals that AWS CloudFormation generates a stream name on your behalf if you don't choose to specify a value.

In addition, notice that Name property of the resource cannot be updated, so specifying a Name that wasn't previously assigned will result in a new resource being created. A special case is when the value is unspecified which will result in AWS CloudFormation generating a unique name for the resource.

Description of Name property (source: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesis-stream.html#cfn-kinesis-stream-name)

Name The name of the Kinesis stream. If you don't specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the stream name. For more information, see Name Type.

If you specify a name, you cannot perform updates that require replacement of this resource. You can perform updates that require no or some interruption. If you must replace the resource, specify a new name.

Required: No

Type: String

Minimum: 1

Maximum: 128

Pattern: [a-zA-Z0-9_.-]+

Update requires: Replacement

The Problem

One of the fundamental benefits of describing an application model in .bicep manifests is that their deployment to a given radius environment is idempotent, however, an operator authoring their AWS infrastructure in Bicep to be consumed by a Radius application developer may inadvertently define the AWS::Kinesis::Stream resource without a Name property since there are no guardrails against it in the Bicep type definition breaking the idempotency promise and resulting in the creation of potentially undesired cloud resources that litter the customers AWS account.

Proposed Solution

The types.json generator for AWS Resources visits the CloudFormation schema to produce the Bicep Extensibility Resource schema (see code section for full details).

We propose to mark all primaryIdentifier and additionalIdentifier properties that are not readOnlyProperties to be flagged as ObjectPropertyFlags.Required. Doing so will restrict the author to specify a value for these properties so that AWS CloudFormation generation logic is never used as a guardrail to prevent creating non-idempotent bicep manifests.

Tasks

[ ] Add the desired logic to flag identifiers as required in convert.ts
[ ] Add unit tests on the new logic above
[ ] Merge and publish new types.json definitions into radbicep repo

Additional Information

In the issue Background we mention that the logic of auto generation of primaryIdentifier properties is a common occurrence, to support this argument we will keep below a running list of resources for which this applies.

More can be found by running the following google query.

Helpful information about CloudFormation field generation behavior - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-name.html AB#4487

radius-project / radius