Unable to update aws-native:pipes:Pipe with Self-Managed Kafka connection due to NotUpdatableException

[valerio-iachini]

What happened?

We are encountering an issue while updating the aws-native:pipes:Pipe resource connected to a self-managed Kafka instance using the PipeSelfManagedKafkaAccessConfigurationCredentials0PropertiesArgs. When we attempt to update the resource, we receive the following error:

error: operation error CloudControl: UpdateResource, https response error StatusCode: 400, RequestID: XXX, NotUpdatableException: Invalid patch update: createOnlyProperties [/properties/SourceParameters/SelfManagedKafkaParameters/TopicName, /properties/SourceParameters/SelfManagedKafkaParameters/StartingPosition] cannot be updated

However, we are not changing the TopicName or StartingPosition properties. We are only updating the version of the enrichment Lambda.

To work around this, we manually updated the Lambda version via the AWS console, refreshed the Pulumi stack, and successfully completed the pulumi up.

Our concern is that this behavior might be a bug in Pulumi, where the tool is attempting to update other properties (like TopicName and StartingPosition) that are designated as createOnlyProperties, instead of only updating the Lambda version as expected.

Could you please investigate whether Pulumi is incorrectly triggering updates to these createOnlyProperties when only the Lambda version is being modified?

Thank you for your support.

Steps to Recreate the Issue:

1. Create the Initial Pipe: Define and deploy the pipe with an initial enrichment Lambda function ARN using pulumi up.

2. Modify the Enrichment ARN: Change the ARN for the enrichment Lambda function in your Pulumi code.

3. Update the Pipe: Run pulumi up again to apply the changes.

Example

from pulumi_aws_native.pipes import (
    Pipe,
    PipeEnrichmentParametersArgs,
    PipeRequestedPipeState,
    PipeSelfManagedKafkaAccessConfigurationCredentials0PropertiesArgs,
    PipeSourceParametersArgs,
    PipeSourceSelfManagedKafkaParametersArgs,
    PipeTargetParametersArgs,
)
pipe_name = "example-pipe-name"
pipe_role_arn = "arn:aws:iam::123456789012:role/example-role" 
kafka_cluster_url = "example-cluster-url"  
queue_arn = "arn:aws:sqs:us-west-2:123456789012:example-queue" 
kafka_secret_arn = "arn:aws:secretsmanager:us-west-2:123456789012:secret:example-secret" 
topic_name = "example-topic" 
enrichment_arn = "arn:aws:lambda:us-west-2:123456789012:function:example-enrichment"
enrichment_input_template = '{"key": "<$.key>", "value": "<$.value>"}'

Pipe(
    f"{pipe_name}",
    name=pipe_name,
    role_arn=pipe_role_arn,
    desired_state=PipeRequestedPipeState.RUNNING,
    source=f"smk://{kafka_cluster_url}",
    target=queue_arn,
    source_parameters=PipeSourceParametersArgs(
        self_managed_kafka_parameters=PipeSourceSelfManagedKafkaParametersArgs(
            batch_size=10,
            credentials=PipeSelfManagedKafkaAccessConfigurationCredentials0PropertiesArgs(
                basic_auth=kafka_secret_arn
            ),
            maximum_batching_window_in_seconds=10,
            starting_position="TRIM_HORIZON",
            topic_name=topic_name,
        ),
    ),
    enrichment=enrichment_arn,
    enrichment_parameters=PipeEnrichmentParametersArgs(
        input_template=enrichment_input_template
    )
)

Output of pulumi about

CLI Version 3.106.0 Go Version go1.22.0 Go Compiler gc

Plugins NAME VERSION aws 5.43.0 aws-native 0.121.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[flostadler]

Hey @valerio-iachini, I'm sorry you're running into this issue!

I was able to reproduce it on our end. Currently the provider is adding all properties marked as writeOnly but not createOnly to the update requests sent to AWS APIs.

For aws-native:pipes:Pipe the sourceParameters parameter is marked as writeOnly and some of the nested properties like sourceParameters/ManagedStreamingKafkaParameters/TopicName are marked as createOnly. Right now the check for writeOnly/createOnly was only operating at the root level, but it seems like it needs to traverse property paths.

As a workaround you could try using the aws-classic provider: https://www.pulumi.com/registry/packages/aws/api-docs/pipes/pipe/

[flostadler]

I checked what other resources are affected by this and the only one that has a similar setup is aws-native.refactorspaces.Route. So this seems to be a fairly isolated issue

[flostadler]

Tracking the underlying issue here: https://github.com/pulumi/pulumi-aws-native/issues/1722