Kusto Database Principal Assignment - Resource partially created but read failed autorest/azure: Service returned an error. Status = 404 Code="ResourceNotFound""

SeanErvinson commented 1 year ago

What happened?

I'm trying to assign a kusto role assignment to a container app and I'm getting this error that seems to happen randomly. Sometimes it works fine and sometimes I have to run it multiple times before it will go through. I've checked that the resource group, database, and cluster exist.

Error message:

error: resource partially created but read failed autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The resource with identifier '/subscriptions/xxx/resourceGroups/rg-dev-00/providers/Microsoft.Kusto/Clusters/dec-dev00/Databases/dedb-dev-00-wil-re1/PrincipalAssignments/dedb-app-admin-digitaltwin-role' is not found.": Code="Failed" Message="Internal Server Error"

I'm wondering if pulumi is hindering the error message similar to this issues

Expected Behavior

Return a more descriptive error or would just create it seamlessly.

Steps to reproduce

This is the code:

    _ = new DatabasePrincipalAssignment("dedb-app-admin-digitaltwin-role",
                                                new DatabasePrincipalAssignmentArgs
                                                {
                                                    ClusterName = "dec-dev00",
                                                    DatabaseName = "dedb-dev-00-wil-re1",
                                                    PrincipalAssignmentName = "dedb-app-admin-digitaltwin-role",
                                                    PrincipalId = "<id of the container app>",
                                                    PrincipalType = "App",
                                                    ResourceGroupName = "rg-dev-00",
                                                    Role = DatabasePrincipalRole.Admin,
                                                },
                                                new CustomResourceOptions
                                                {
                                                    Parent = this,
                                                    Provider = new Pulumi.AzureNative.Provider($"cluster-azure-provider", new Pulumi.AzureNative.ProviderArgs { SubscriptionId = "<subscription id of where the cluster resides>" }),
                                                });

Output of `pulumi about`

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:19.24

'dotnet build -nologo .' completed successfully
warning: Failed to get information about the current stack: No current snapshot
CLI          
Version      3.76.1
Go Version   go1.20.6
Go Compiler  gc

Plugins
NAME          VERSION
azure-native  2.2.0
azuread       5.40.0
dotnet        unknown
random        4.13.2

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in dotnet: executable='/usr/share/dotnet/dotnet' version='6.0.412'

Backend        
Name           fv-az306-578
URL            file://~
User           runner
Organizations  

Dependencies:
NAME    VERSION
Dapper  2.0.143

Pulumi locates its logs in /tmp by defaultBuild succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:19.24

'dotnet build -nologo .' completed successfully
warning: Failed to get information about the current stack: No current snapshot
CLI          
Version      3.76.1
Go Version   go1.20.6
Go Compiler  gc

Plugins
NAME          VERSION
azure-native  2.2.0
azuread       5.40.0
dotnet        unknown
random        4.13.2

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in dotnet: executable='/usr/share/dotnet/dotnet' version='6.0.412'

Backend        
Name           fv-az306-578
URL            file://~
User           runner
Organizations  

Dependencies:
NAME    VERSION
Dapper  2.0.143

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

danielrbradley commented 1 year ago

Hi @SeanErvinson thanks for reporting this issue.

Does the DatabasePrincipalAssignment resource creation eventually complete and appear in the Azure portal or does the create actually fail?

Are you able to locate any activity logs from within the Azure portal which you could redact and share?

SeanErvinson commented 1 year ago

Hi @danielrbradley , the resource didn't get created. Here is what I got from azure

{
    "authorization": {
        "action": "Microsoft.Kusto/clusters/databases/principalAssignments/write",
        "scope": "/subscriptions/<subid>/resourceGroups/rg-dev-00/providers/Microsoft.Kusto/clusters/dec-dev00/databases/dedb-dev-00-wil-re1/principalAssignments/dedb-app-admin-digitaltwin-role"
    },
    "caller": "bd2c9873-feb2-4948-88c6-257544aecb64",
    "channels": "Operation",
    "claims": {
        // redacted
    },
    "correlationId": "2cb72994-1d14-4b73-b6b3-c3029d6baeec",
    "description": "",
    "eventDataId": "58fab150-f563-4093-8597-0a406396313c",
    "eventName": {
        "value": "EndRequest",
        "localizedValue": "End request"
    },
    "category": {
        "value": "Administrative",
        "localizedValue": "Administrative"
    },
    "eventTimestamp": "2023-08-04T03:28:39.3354609Z",
    "id": "/subscriptions/<subid>/resourceGroups/rg-dev-00/providers/Microsoft.Kusto/clusters/dec-dev00/databases/dedb-dev-00-wil-re1/principalAssignments/dedb-app-admin-digitaltwin-role/events/58fab150-f563-4093-8597-0a406396313c/ticks/638267165193354609",
    "level": "Error",
    "operationId": "92a720c4-34ed-42ef-abf3-bd1744fd9d08",
    "operationName": {
        "value": "Microsoft.Kusto/clusters/databases/principalAssignments/write",
        "localizedValue": "Write a database principal assignments resource"
    },
    "resourceGroupName": "rg-dev-00",
    "resourceProviderName": {
        "value": "Microsoft.Kusto",
        "localizedValue": "Microsoft.Kusto"
    },
    "resourceType": {
        "value": "Microsoft.Kusto/clusters/databases/principalAssignments",
        "localizedValue": "Microsoft.Kusto/clusters/databases/principalAssignments"
    },
    "resourceId": "/subscriptions/<subid>/resourceGroups/rg-dev-00/providers/Microsoft.Kusto/clusters/dec-dev00/databases/dedb-dev-00-wil-re1/principalAssignments/dedb-app-admin-digitaltwin-role",
    "status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "",
        "localizedValue": ""
    },
    "submissionTimestamp": "2023-08-04T03:31:31Z",
    "subscriptionId": "<subid>",
    "tenantId": "d43166d1-c2a1-4f26-a213-f620dba13ab8",
    "properties": {
        "statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"Failed\",\"message\":\"Internal Server Error\"}]}}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/<subid>/resourceGroups/rg-dev-00/providers/Microsoft.Kusto/clusters/dec-dev00/databases/dedb-dev-00-wil-re1/principalAssignments/dedb-app-admin-digitaltwin-role",
        "message": "Microsoft.Kusto/clusters/databases/principalAssignments/write",
        "hierarchy": "d43166d1-c2a1-4f26-a213-f620dba13ab8/WillowTwinDev/<subid>"
    },
    "relatedEvents": []
}

danielrbradley commented 1 year ago

Ah that's perfect thank you @SeanErvinson!

The Internal Server Error in the status message indicates to me that this is a bug in the Azure service implementation. If there was an issue with the inputs our provider had sent then the error message would have indicated that and I'd expect that to be surfaced back through the provider.

I'd suggest raising a support ticket with Azure referencing that event to get them to investigate why the creation failed with a nondescript server-side error message.

Please do let us know if there's any relevant details to our provider. If we haven't heard back in the next couple of months we'll close out the issue as stale with the hope Azure has fixed this their side. We'll leave open for now in case any other users are hitting the same issue.

pulumi / pulumi-azure-native