Metastore assignment appears incorrect with multiple metastores present

james-servco commented 6 months ago

What happened?

We have a databricks environment with multiple metastores (one that will soon be deprecated, one that we're moving to) When I assign a new workspace to the new metastore, it appears to successfully assign it the first time it's created, but on subsequent pulumi up's, it believes that the metastore assignment metastoreId is the old metastore (which is also has never been assigned to). If i proceed with this change, it appears to remove the assignment off the workspace completely.

Example

az_databricks_workspace = az_databricks.Workspace("dbw-default",
    resource_group_name=resource_group.name,
    location=azure_location,
    sku=az_databricks.SkuArgs(
        name='Premium',
    ),
    managed_resource_group_id=f"/subscriptions/{sub_id}/resourceGroups/rg-dataprojects-databricks",
    workspace_name=databricksworkspacename,
    opts=pulumi.ResourceOptions(parent=resource_group)
)

# Assign the existing Metastore to the new Databricks workspace
metastore_assignment = databricks.MetastoreAssignment("dma-default",
    workspace_id=az_databricks_workspace.workspace_id,
    metastore_id=metastore_id,
    default_catalog_name="main",
    opts=pulumi.ResourceOptions(
        parent=az_databricks_workspace
    )
)

Output of `pulumi about`

CLI
Version 3.107.0 Go Version go1.22.0 Go Compiler gc

Plugins NAME VERSION azure-native 2.29.0 databricks 1.32.2 python unknown random 4.15.1

Host
OS darwin Version 14.2.1 Arch arm64

This project is written in python: executable='/opt/homebrew/bin/python3' version='3.11.7'

Backend
Name pulumi.com URL https://app.pulumi.com/jw-servco User jw-servco Organizations jw-servco, servco Token type personal

Dependencies: NAME VERSION databricks 0.2 pip 24.0 pulumi_azure_native 2.29.0 pulumi_databricks 1.32.2 pulumi_random 4.15.1 setuptools 69.1.0 wheel 0.42.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

iwahbe commented 6 months ago

Hi @james-servco. I'm sorry you're hitting this issue. Just to clarify, the first time you run pulumi up it works as expected, but then subsequent pulumi ups produce invalid results. Is that right?

Could you give us a self-contained repro that we can run locally.

Looks like this may be related to https://github.com/databricks/terraform-provider-databricks/issues/2489.

james-servco commented 6 months ago

Hey! Yep that is correct. The initial pulumi up works as expected, but subsequent ones (once the resources are created) report that the metastoreId points to the other metastore that's never been assigned, not the one that is currently assigned to it.

I'll attempt to create a repro - but i suspect the issue mentioned appears to be in line with what's happening!

pulumi / pulumi-databricks