search

pulumi / pulumi-azure-native

Azure Native Provider
Apache License 2.0
123 stars 32 forks source link

dbforpostgressql Encryption with CMK #3403

Closed AngelShresth1 closed 1 day ago

AngelShresth1 commented 1 week ago

What happened?

Hello, I tried to create Azure DB for PostgresSQL with Data Encryption but ran into this error.

azure-native:dbforpostgresql:Server (server5): error: Code="InternalServerError" Message="An unexpected error occured while processing the request. Tracking ID: '448e0c16-b379-4742-861a-36ae91a3c6df'"

Code used

server5 = azure_native.dbforpostgresql.Server("server5",
    administrator_login='tuvaadmin',
    administrator_login_password='tuv1pg1!',
    backup=azure_native.dbforpostgresql.BackupArgs(
        backup_retention_days=7,
        geo_redundant_backup=azure_native.dbforpostgresql.GeoRedundantBackupEnum.DISABLED,
    ),
    location=resource_group_location,
    resource_group_name=resource_group.name,
    server_name=f"uh-{client_name}-server5",
    sku=azure_native.dbforpostgresql.SkuArgs(
        name="Standard_B1ms",
        tier=azure_native.dbforpostgresql.SkuTier.BURSTABLE,
    ),
    storage=azure_native.dbforpostgresql.StorageArgs(
        storage_size_gb=32,
    ),
    network=azure_native.dbforpostgresql.NetworkArgs(
            delegated_subnet_resource_id=f"/subscriptions/{subscription_id}/resourceGroups/{resource_group_name}/providers/Microsoft.Network/virtualNetworks/{vnet_name}/subnets/{db_subnet_name}",
            private_dns_zone_arm_resource_id=f"/subscriptions/{subscription_id}/resourceGroups/{resource_group_name}/providers/Microsoft.Network/privateDnsZones/{client_name}.[private.postgres.database.azure.com](http://private.postgres.database.azure.com/)",
        ),
    identity=azure_native.dbforpostgresql.UserAssignedIdentityArgs(
             type='UserAssigned',
        user_assigned_identities={
            f"/subscriptions/{subscription_id}/resourcegroups/{resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{db_user_assigned_identity_name}" : azure_native.dbforpostgresql.UserIdentityArgs()},
    ),
    data_encryption=azure_native.dbforpostgresql.DataEncryptionArgs(
            type="AzureKeyVault",
            primary_key_uri=f"https://{vault_name}.[vault.azure.net/keys/{key.name}/{key_version}](http://vault.azure.net/keys/%7Bkey.name%7D/%7Bkey_version%7D)",
            primary_user_assigned_identity_id=f"/subscriptions/{subscription_id}/resourcegroups/{resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{db_user_assigned_identity_name}",
        ),
    tags={
        "created_by": author,
    },
    version=azure_native.dbforpostgresql.ServerVersion.SERVER_VERSION_12,
    opts=pulumi.ResourceOptions(depends_on=[
                # db_subnet, private_db_dns_zone, db_nsg, vault, db_user_assigned_identity, key, resource_group, storage_subnet,aks_cluster,vnet
                db_subnet, private_db_dns_zone, vault, db_user_assigned_identity, key, resource_group, vnet,aks_cluster
            ],))

Example

` server5 = azure_native.dbforpostgresql.Server("server5", administrator_login='tuvaadmin', administrator_login_password='tuv1pg1!', backup=azure_native.dbforpostgresql.BackupArgs( backup_retention_days=7, geo_redundant_backup=azure_native.dbforpostgresql.GeoRedundantBackupEnum.DISABLED, ), location=resource_group_location, resource_group_name=resource_group.name, server_name=f"uh-{client_name}-server5", sku=azure_native.dbforpostgresql.SkuArgs( name="Standard_B1ms", tier=azure_native.dbforpostgresql.SkuTier.BURSTABLE, ), storage=azure_native.dbforpostgresql.StorageArgs( storage_size_gb=32, ), network=azure_native.dbforpostgresql.NetworkArgs( delegated_subnet_resource_id=f"/subscriptions/{subscription_id}/resourceGroups/{resource_group_name}/providers/Microsoft.Network/virtualNetworks/{vnet_name}/subnets/{db_subnet_name}", private_dns_zone_arm_resource_id=f"/subscriptions/{subscription_id}/resourceGroups/{resource_group_name}/providers/Microsoft.Network/privateDnsZones/{client_name}.private.postgres.database.azure.com", ), identity=azure_native.dbforpostgresql.UserAssignedIdentityArgs( type='UserAssigned', user_assigned_identities={ f"/subscriptions/{subscription_id}/resourcegroups/{resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{db_user_assigned_identity_name}" : azure_native.dbforpostgresql.UserIdentityArgs()}, ), data_encryption=azure_native.dbforpostgresql.DataEncryptionArgs( type="AzureKeyVault", primary_key_uri=f"https://{vault_name}.vault.azure.net/keys/{key.name}/{key_version}", primary_user_assigned_identity_id=f"/subscriptions/{subscription_id}/resourcegroups/{resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{db_user_assigned_identity_name}", ),

tags={
    "created_by": author,
},

version=azure_native.dbforpostgresql.ServerVersion.SERVER_VERSION_12,
opts=pulumi.ResourceOptions(depends_on=[
            # db_subnet, private_db_dns_zone, db_nsg, vault, db_user_assigned_identity, key, resource_group, storage_subnet,aks_cluster,vnet
            db_subnet, private_db_dns_zone, vault, db_user_assigned_identity, key, resource_group, vnet,aks_cluster

        ],))`

Output of pulumi about

image

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

thomas11 commented 1 week ago

Hi @AngelShresth1, sorry you're running into trouble here. "Internal server error" suggests a bug on the Azure side but we can still try to see if we can fix it on our side.

First, we'd need to know the payload the Pulumi provider sends to Azure. Could you run your program with log level 9? Then, if you don't want to upload the whole log here, you can search for the lines starting with HTTP Response Begin PUT.

I had a quick look at running your code, but it's only the server resource and lacks several dependent resources like vnet, key, identity etc.

AngelShresth1 commented 5 days ago

11:28:50.942935 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>===================================================== HTTP Response End PUT https://management.azure.com/subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/uh-baseline2-server?api-version=2022-12-01 <{%reset%}>) I0705 11:28:52.893828 1439415 api.go:314] Making Pulumi API call: https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/events/batch I0705 11:28:53.510601 1439415 api.go:328] Pulumi API call response code (https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/events/batch): 200 OK I0705 11:30:44.063337 1439415 api.go:314] Making Pulumi API call: https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/renew_lease I0705 11:30:46.247361 1439415 api.go:328] Pulumi API call response code (https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/renew_lease): 200 OK I0705 11:30:50.942919 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>I0705 11:30:50.942835 1439841 client.go:441] HTTP Request Begin PUT https://management.azure.com/subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/uh-baseline2-server?api-version=2022-12-01 =================================================== <{%reset%}>) I0705 11:30:50.943018 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>PUT /subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/uh-baseline2-server?api-version=2022-12-01 HTTP/1.1 <{%reset%}>) I0705 11:30:50.943042 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Host: management.azure.com <{%reset%}>) I0705 11:30:50.943056 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>User-Agent: Go/go1.22.4 (amd64-linux) go-autorest/v14.2.1 Go/go1.22.4 (amd64-linux) go-autorest/v14.2.1 pulumi-azure-native/2.47.0 pid-a90539d8-a7a6-5826-95c4-1fbef22d4b22 <{%reset%}>) I0705 11:30:50.943069 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Content-Length: 1821 <{%reset%}>) I0705 11:30:50.943089 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Authorization: Bearer cmVzdGhhQHVudmVpbC5oZWFsdGgiLCJ1dGkiOiJZRDNOM1lfWWJFV0tpQWxvMWdReEFBIiwidmVyIjoiMS4wIiwid2lkcyI6WyJiNzlmYmY0ZC0zZWY5LTQ2ODktODE0My03NmIxOTRlODU1MDkiXSwieG1zX2NhZSI6IjEiLCJ4bXNfY2MiOlsiQ1AxIl0sInhtc19maWx0ZXJfaW5kZXgiOlsiNzAiXSwieG1zX2lkcmVsIjoiMSAyMiIsInhtc19yZCI6IjAuNDJMbFlCUmlkQU1BIiwieG1zX3NzbSI6IjEiLCJ4bXNfdGNkdCI6MTY4MTMyNzc0NX0.eyTv_nupLBn1FguSQPXINtpjplibfDiLnxHK438T5F6TEMLKkoLq3N2n_R3P0jX-rSgDT1UpW0fyO_GDLoB7EmM5x8KpAtRS6amRPpo2g56_fguseJSPRqfsZ7oqJjYRP4yQEjvyo84rCyn-iNrgnZQTVEQU26i9g_Xii32c3MJgdfifuUKGVRwy4o11kH4-uxjIRnrP6n-E4u-zOs2CdYNza3Ii_iSq-k9VjSJTzL6F3cYGoG9L_yRKSjUNWSJp0ryCSWgfPvrVBU0_GYFFEb7c1gDV4mufWgnm4rSntKzBSMf8SNlka7Vm-vaLr4m6Mj5Yi8xMia3UeAIfz8vV9Q <{%reset%}>) I0705 11:30:50.943107 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Content-Type: application/json; charset=utf-8 <{%reset%}>) I0705 11:30:50.943139 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>{"identity":{"type":"UserAssigned","userAssignedIdentities":{"/subscriptions/subscription_id/resourcegroups/cbva-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cbva-db-useridentity":{}}},"location":"eastus","properties":{"administratorLogin":"tuvaadmin","administratorLoginPassword":"tuv1pg1!","availabilityZone":"","backup":{"backupRetentionDays":7,"geoRedundantBackup":"Disabled"},"dataEncryption":{"primaryKeyURI":"https://cbva-keyvault-76.vault.azure.net/keys/key/version.","primaryUserAssignedIdentityId":"/subscriptions/subscription_id/resourcegroups/cbva-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cbva-db-useridentity","type":"AzureKeyVault"},"network":{"delegatedSubnetResourceId":"/subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.Network/virtualNetworks/cbva_dev_backstage/subnets/cbva_psql_subnet2","privateDnsZoneArmResourceId":"/subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.Network/privateDnsZones/cbva.private.postgres.database.azure.com"},"storage":{"storageSizeGB":32},"version":"12"},"sku":{"name":"Standard_B1ms","tier":"Burstable"},"tags":{"created_by":"angel"}} <{%reset%}>) I0705 11:30:50.943167 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>===================================================== HTTP Request End PUT https://management.azure.com/subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/uh-baseline2-server?api-version=2022-12-01 <{%reset%}>) I0705 11:30:52.894424 1439415 api.go:314] Making Pulumi API call: https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/events/batch I0705 11:30:53.236436 1439415 api.go:328] Pulumi API call response code (https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/events/batch): 200 OK I0705 11:30:53.708881 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>I0705 11:30:53.708785 1439841 client.go:464] HTTP Response Begin PUT [https://management.azure.com/subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/uh-baseline2-server?api-version=2022-12-01 =================================================== <{%reset%}>) I0705 11:30:53.708952 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>HTTP/2.0 500 Internal Server Error <{%reset%}>) I0705 11:30:53.708973 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Content-Length: 162 <{%reset%}>) I0705 11:30:53.708982 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Cache-Control: no-cache <{%reset%}>) I0705 11:30:53.708992 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Content-Type: application/json; charset=utf-8 <{%reset%}>) I0705 11:30:53.708998 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Date: Fri, 05 Jul 2024 11:30:53 GMT <{%reset%}>) I0705 11:30:53.709005 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Expires: -1 <{%reset%}>) I0705 11:30:53.709014 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Pragma: no-cache <{%reset%}>) I0705 11:30:53.709027 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Strict-Transport-Security: max-age=31536000; includeSubDomains <{%reset%}>) I0705 11:30:53.709052 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Cache: CONFIG_NOCACHE <{%reset%}>) I0705 11:30:53.709061 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Content-Type-Options: nosniff <{%reset%}>) I0705 11:30:53.709080 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Ms-Correlation-Request-Id: f6cf5885-b0b6-43d6-952a-8a6faf6c3057 <{%reset%}>) I0705 11:30:53.709110 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Ms-Failure-Cause: service <{%reset%}>) I0705 11:30:53.709124 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Ms-Ratelimit-Remaining-Subscription-Global-Writes: 2999 <{%reset%}>) I0705 11:30:53.709134 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Ms-Ratelimit-Remaining-Subscription-Writes: 199 <{%reset%}>) I0705 11:30:53.709140 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Ms-Request-Id: 2d802704-7dcb-4f3e-8f3d-509eba7eb36a <{%reset%}>) I0705 11:30:53.709147 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Ms-Routing-Request-Id: SOUTHINDIA:20240705T113053Z:f6cf5885-b0b6-43d6-952a-8a6faf6c3057 <{%reset%}>) I0705 11:30:53.709154 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>X-Msedge-Ref: Ref A: 69F94E5953E94868BF1844CBF17C61D7 Ref B: MAA201060516039 Ref C: 2024-07-05T11:30:51Z <{%reset%}>) I0705 11:30:53.709163 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>{"error":{"code":"InternalServerError","message":"An unexpected error occured while processing the request. Tracking ID: '2d802704-7dcb-4f3e-8f3d-509eba7eb36a'"}} <{%reset%}>) I0705 11:30:53.709177 1439415 eventsink.go:78] eventSink::Infoerr(<{%reset%}>===================================================== HTTP Response End PUT https://management.azure.com/subscriptions/subscription_id/resourceGroups/cbva-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/uh-baseline2-server?api-version=2022-12-01 <{%reset%}>) I0705 11:30:56.895212 1439415 api.go:314] Making Pulumi API call: https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/events/batch I0705 11:30:57.703908 1439415 api.go:328] Pulumi API call response code (https://api.pulumi.com/api/stacks/AngelShresth1/StructuredPulumi/1azure/update/38e48d7c-7346-4762-9d6a-a9b665cbfaa9/events/batch): 200 OK

Hello @thomas11 , thank you for initiation, this is the logs

Also , when i use same resources like vnet, key and identity and create postgres from portal it is easily created an when I comment out the data_encryption=azure_native.dbforpostgresql.DataEncryptionArgs( type="AzureKeyVault", primary_key_uri=f"https://{vault_name}.vault.azure.net/keys/{key.name}/{key_version}", primary_user_assigned_identity_id=f"/subscriptions/{subscription_id}/resourcegroups/{resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{db_user_assigned_identity_name}", ), data encryption aprt then also the server is created

thomas11 commented 1 day ago

Hi @AngelShresth1 , I think there's a problem with the way the KV key is specified. In the request sent to Azure, it looks like this:

"primaryKeyURI": "https://cbva-keyvault-76.vault.azure.net/keys/key/version.",

Note the "version." at the end instead of the actual version. Can you double-check the key_version variable? Should it maybe be key.version?

AngelShresth1 commented 1 day ago

Thanks @thomas11 , it was indeed the key_version, I updated it and now it can easily create the server