search

pulumi / pulumi-aws-native

AWS Native Provider for Pulumi
Apache License 2.0
95 stars 17 forks source link

aws-native:datazone:DataSource: Create Datasource fails with "2 subschemas matched instead of one" #1846

Closed MeTimesThree closed 1 day ago

MeTimesThree commented 4 days ago

What happened?

We receive a strange error when trying to create a new Datazone-DataSource.

We see: ValidationException: Model validation failed (#/Configuration: #: 2 subschemas matched instead of one

In the debug-log we see a succesful Unmarshaling: I1122 12:56:31.903150 32148 rpc.go:292] Unmarshaling property for RPC[aws-native.Create(urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:DataSource::kk_source_redshift-poc-dp).properties]: configuration={map[redshiftRunConfiguration:{map[dataAccessRole:{arn:aws:iam::381492292231:role/datazone-redshift-manage-access-role-poc-dpServRole} redshiftCredentialConfiguration:{map[secretManagerArn:{arn:aws:secretsmanager:eu-central-1:381492292231:secret:kk_redshift_credentials202411212-poc-dp-StKPm2}]} redshiftStorage:{map[redshiftClusterSource:{map[clusterName:{redshift-poc-dp}]}]} relationalFilterConfigurations:{[{map[databaseName:{dwh_poc_db} filterExpressions:{[{map[expression:{*} type:{INCLUDE}]}]} schemaName:{public}]}]}]}]}

But then a DesiredState with a seemingly empty configuration appears in the log: {"ClientToken":"fb7c17f6-ccc3-4b39-8bb7-c2f67e920ebb","DesiredState":"{\"Configuration\":{},\"Description\":\"KK DataSource des PoC\",\"DomainIdentifier\":\"dzd_arcll8lb4xk7bf\",\"EnvironmentIdentifier\":\"boyub3bhivj4ln\",\"Name\":\"kk_source_redshift-poc-dp\",\"ProjectIdentifier\":\"4bdmaeh6uinsej\",\"PublishOnImport\":true,\"Recommendation\":{\"EnableBusinessNameGeneration\":false},\"Type\":\"REDSHIFT\"}","TypeName":"AWS::DataZone::DataSource"}

Example

This is the Pulumi-main that fails: Pulumi-Main It needs the following dependency: SftSecurityGroup

The CLI works fine with the following commands: AWS CLI Call

Output of pulumi about

CLI          
Version      3.140.0
Go Version   go1.23.3
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  aws         6.60.0
resource  aws-native  1.9.0
language  python      unknown
resource  std         1.6.2
resource  str         1.0.0

Host     
OS       fedora
Version  40
Arch     x86_64

This project is written in python: executable='/home/u000451/repos/sft-bi-poc/pulumi/datenkatalog/venv/bin/python' version='3.12.7'

Current Stack: organization/datenkatalog/datenkatalog

TYPE                                                       URN
pulumi:pulumi:Stack                                        urn:pulumi:datenkatalog::datenkatalog::pulumi:pulumi:Stack::datenkatalog-datenkatalog
pulumi:providers:aws                                       urn:pulumi:datenkatalog::datenkatalog::pulumi:providers:aws::default_6_60_0
aws:ec2/vpc:Vpc                                            urn:pulumi:datenkatalog::datenkatalog::aws:ec2/vpc:Vpc::vpc-poc-dp
aws:iam/role:Role                                          urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::Redshift-poc-dpServRole
aws:iam/role:Role                                          urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::datazone-domain-execution-role-poc-dpServRole
components:index:SftSecurityGroup                          urn:pulumi:datenkatalog::datenkatalog::components:index:SftSecurityGroup::sftSecurityGroupRedshift
aws:ec2/subnet:Subnet                                      urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_private_1-poc-dp
pulumi:providers:aws-native                                urn:pulumi:datenkatalog::datenkatalog::pulumi:providers:aws-native::default_1_9_0
aws:ec2/subnet:Subnet                                      urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_private_2-poc-dp
aws:ec2/securityGroup:SecurityGroup                        urn:pulumi:datenkatalog::datenkatalog::aws:ec2/securityGroup:SecurityGroup::sftSecurityGroupRedshift-sft_security_group
aws:redshift/subnetGroup:SubnetGroup                       urn:pulumi:datenkatalog::datenkatalog::aws:redshift/subnetGroup:SubnetGroup::sub_group_redshift-poc-dp
aws:vpc/securityGroupEgressRule:SecurityGroupEgressRule    urn:pulumi:datenkatalog::datenkatalog::aws:vpc/securityGroupEgressRule:SecurityGroupEgressRule::sftSecurityGroupRedshift-sft_security_group_all_outgoing
aws-native:datazone:Domain                                 urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:Domain::datazone_domain_bank-poc-dp
aws:vpc/securityGroupIngressRule:SecurityGroupIngressRule  urn:pulumi:datenkatalog::datenkatalog::aws:vpc/securityGroupIngressRule:SecurityGroupIngressRule::sftSecurityGroupRedshift-sft_security_group_self_referincing
aws:iam/role:Role                                          urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::datazone-provisioning-role-poc-dpServRole
aws:iam/role:Role                                          urn:pulumi:datenkatalog::datenkatalog::aws:iam/role:Role::datazone-redshift-manage-access-role-poc-dpServRole
aws-native:datazone:Project                                urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:Project::datazone_project_kk-poc-dp
aws:ec2/subnet:Subnet                                      urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_public-poc-dp
aws:ec2/subnet:Subnet                                      urn:pulumi:datenkatalog::datenkatalog::aws:ec2/subnet:Subnet::subnet_public_2-poc-dp
aws-native:datazone:EnvironmentBlueprintConfiguration      urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:EnvironmentBlueprintConfiguration::datazone_bank_blup_config_redshift-poc-dp
aws-native:datazone:EnvironmentProfile                     urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:EnvironmentProfile::kk_datazone_bank_env_profile_redshift-poc-dp
aws:redshift/cluster:Cluster                               urn:pulumi:datenkatalog::datenkatalog::aws:redshift/cluster:Cluster::redshift_kernbank-poc-dp
aws:secretsmanager/secret:Secret                           urn:pulumi:datenkatalog::datenkatalog::aws:secretsmanager/secret:Secret::kk_redshift_credentials
aws:secretsmanager/secretVersion:SecretVersion             urn:pulumi:datenkatalog::datenkatalog::aws:secretsmanager/secretVersion:SecretVersion::kk_redshift_credentials_version
aws-native:datazone:Environment                            urn:pulumi:datenkatalog::datenkatalog::aws-native:datazone:Environment::kk_datazone_bank_env_redshift-poc-dp

Found no pending operations associated with datenkatalog

Backend        
Name           fedora.fritz.box
URL            s3://pulumi-state-bic-poc
User           u000451
Organizations  
Token type     personal

Dependencies:
NAME               VERSION
pip                24.3.1
pulumi_aws         6.60.0
pulumi_aws_native  1.9.0
pulumi_std         1.6.2
pulumi_str         1.0.0
setuptools         75.2.0
wheel              0.44.0

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

t0yv0 commented 1 day ago

Thank you for an excellent repro, this is indeed a bug in the provider. I have narrowed it down to this spot:

https://github.com/pulumi/pulumi-aws-native/blob/eb0829ba5a2aa6da9a644791631c35f29f3aabdd/provider/pkg/naming/convert.go#L96

The type of "configuration" is a union of possibilities and this code incorrectly selects option 1 instead of option 2.

t0yv0 commented 1 day ago

I filed https://github.com/pulumi/pulumi-aws-native/issues/1849 for the root cause.