flostadler
commented
7 hours ago So sorry you're running into this issue @MeTimesThree!
Both the ProjectIdentifier and EnvironmentIdentifier are createOnly properties in AWS CloudControl (which aws-native uses under the hood).
Our assumption is that those mustn't be sent as part of update requests: https://github.com/pulumi/pulumi-aws-native/blob/417ae358943f688109b41efd061caf897cb1176f/provider/pkg/resources/patching.go#L19-L22
I'll try to replicate this with both pulumi and AWS CloudControl directly to find the root cause.
What happened?
The update of a Datazone-DataSource (in this case the enableBusinessNameGeneration-property) fails with the following error:
` error: operation error CloudControl: UpdateResource, https response error StatusCode: 400, RequestID: 12dd0157-c9e6-46ba-b168-f2146b452bd1, api error ValidationException: Model validation failed (#: required key [DomainIdentifier] not found
: required key [ProjectIdentifier] not found
`
In CloudTrail we see the following request-parameters:
"requestParameters": { "typeName": "AWS::DataZone::DataSource", "clientToken": "<redacted>", "identifier": "<domain>|49ngwew1svuydn", "patchDocument": "HIDDEN_DUE_TO_SECURITY_REASONS" },This is the patchDocument from the pulumi-debug:
pulumi:pulumi:Stack datenkatalog-datenkatalog running {"ClientToken":"<redacted>","Identifier":"<domain>|49ngwew1svuydn","PatchDocument":"[{\"op\":\"add\",\"path\":\"/Configuration\",\"value\":{\"GlueRunConfiguration\":{\"AutoImportDataQualityResult\":false,\"DataAccessRole\":\"arn:aws:iam::381492292231:role/datazone-glue-manage-access-role-poc-dpServRole\",\"RelationalFilterConfigurations\":[{\"DatabaseName\":\"glue-poc-db\",\"FilterExpressions\":[{\"Expression\":\"kooperationspartner\",\"Type\":\"INCLUDE\"}]}]}}},{\"op\":\"replace\",\"path\":\"/Recommendation\",\"value\":{\"EnableBusinessNameGeneration\":false}}]","TypeName":"AWS::DataZone::DataSource"}Sadly i have no further ideas on how to debug this but i will happily assist in further debugging!
Example
This is the Pulumi-main that fails: It needs the following dependency: SftSecurityGroup
You should just be able to "Pulumi up" without issues and can then change enable_business_name_generation to true in line 459. The next "Pulumi up" should produce the error.
Output of
pulumi aboutCLI
Version 3.141.0 Go Version go1.23.3 Go Compiler gc
Plugins KIND NAME VERSION resource aws 6.61.0 resource aws-native 1.10.0 language python unknown resource std 1.6.2 resource str 1.0.0
Host
OS fedora Version 40 Arch x86_64
This project is written in python: executable='/home/u000451/repos/sft-bi-poc/pulumi/datenkatalog/venv/bin/python' version='3.12.7'
Current Stack: organization/datenkatalog/datenkatalog
TYPE URN pulumi:pulumi:Stack urn:pulumi:datenkatalog::datenkatalog::pulumi:pulumi:Stack::datenkatalog-datenkatalog pulumi:providers:aws urn:pulumi:datenkatalog::datenkatalog::pulumi:providers:aws::default_6_61_0 aws:ec2/vpc:Vpc urn:pulumi:datenkatalog::datenkatalog::aws:ec2/vpc:Vpc::vpc-poc-dp aws:iam/role:Role urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::Redshift-poc-dpServRole aws:ec2/subnet:Subnet urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_private_1-poc-dp components:index:SftSecurityGroup urn:pulumi:datenkatalog::datenkatalog::components:index:SftSecurityGroup::sftSecurityGroupRedshift aws:ec2/subnet:Subnet urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_public-poc-dp aws:ec2/securityGroup:SecurityGroup urn:pulumi:datenkatalog::datenkatalog::aws:ec2/securityGroup:SecurityGroup::sftSecurityGroupRedshift-sft_security_group aws:ec2/subnet:Subnet urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_private_2-poc-dp aws:ec2/subnet:Subnet urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_public_2-poc-dp aws:vpc/securityGroupEgressRule:SecurityGroupEgressRule urn:pulumi:datenkatalog::datenkatalog::aws:vpc/securityGroupEgressRule:SecurityGroupEgressRule::sftSecurityGroupRedshift-sft_security_group_all_outgoing aws:vpc/securityGroupIngressRule:SecurityGroupIngressRule urn:pulumi:datenkatalog::datenkatalog::aws:vpc/securityGroupIngressRule:SecurityGroupIngressRule::sftSecurityGroupRedshift-sft_security_group_self_referincing aws:redshift/subnetGroup:SubnetGroup urn:pulumi:datenkatalog::datenkatalog::aws:redshift/subnetGroup:SubnetGroup::sub_group_redshift-poc-dp aws:redshift/cluster:Cluster urn:pulumi:datenkatalog::datenkatalog::aws:redshift/cluster:Cluster::redshift_kernbank-poc-dp aws:iam/role:Role urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::datazone-domain-execution-role-poc-dpServRole pulumi:providers:aws-native urn:pulumi:datenkatalog::datenkatalog::pulumi:providers:aws-native::default_1_10_0 aws-native:datazone:Domain urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:Domain::datazone_domain_bank-poc-dp aws:iam/role:Role urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::datazone-redshift-manage-access-role-poc-dpServRole aws:iam/role:Role urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::datazone-provisioning-role-poc-dpServRole aws-native:datazone:Project urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:Project::datazone_project_kk-poc-dp aws-native:datazone:EnvironmentBlueprintConfiguration urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:EnvironmentBlueprintConfiguration::datazone_bank_blup_config_redshift-poc-dp aws:secretsmanager/secret:Secret urn:pulumi:datenkatalog::datenkatalog::aws:secretsmanager/secret:Secret::kk_redshift_credentials aws-native:datazone:EnvironmentProfile urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:EnvironmentProfile::kk_datazone_bank_env_profile_redshift-poc-dp aws:secretsmanager/secretVersion:SecretVersion urn:pulumi:datenkatalog::datenkatalog::aws:secretsmanager/secretVersion:SecretVersion::kk_redshift_credentials_version aws-native:datazone:Environment urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:Environment::kk_datazone_bank_env_redshift-poc-dp aws-native:datazone:DataSource urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:DataSource::kk_source_redshift-poc-dp
Found no pending operations associated with datenkatalog
Backend
Name fedora.fritz.box URL s3://pulumi-state-bic-poc User u000451 Organizations
Token type personal
Dependencies: NAME VERSION pandas 2.2.3 pip 24.3.1 pulumi_aws 6.61.0 pulumi_aws_native 1.10.0 pulumi_std 1.6.2 pulumi_str 1.0.0 setuptools 75.2.0 wheel 0.44.0
Pulumi locates its logs in /tmp by default
Additional context
We were able to circumvent the error by adding the following to the datasource (to delete and recreate it):
opts = pulumi.ResourceOptions(replace_on_changes=["*"], delete_before_replace=True),However, if you add Subscriptions in Datazone, you cannot delete the DataSource anymore, so sadly that is not a workaround if the DataSource is used.Contributing
Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).