Unable to get past the errors when using the import resource option

t0yv0 commented 3 months ago

What happened?

Seeing customer reports that for certain bridged providers using the import resource option does not work as expected and the user is unable to get past "inputs to import do not match the existing resource", while the provider author is unable to make a fix to make the problem for user to go away.

In particular:

editing the program to match the imported inputs not help
using ignoreChanges does not help

Example

We do not yet have an example with an actual Pulumi-bridged provider. There is a synthetic example in #2216 that uses Plugin Framework to define a resource with a Default value. Customer is reporting this issue in a private bridged provider we cannot reference here.

Output of `pulumi about`

pulumi 3.124.0 bridge 3.88.0

Additional context

There are some related issues.

https://github.com/pulumi/pulumi/issues/3737 would be nice to have so the user at least is presented with more info on which inputs are not matching without having to run with details. It would not fix the issue though.

https://github.com/pulumi/pulumi/issues/16397 suggests a desire to overcome the "inputs to import do not match the existing resource" error - if that had a solution it could be valuable here, if the user could at least workaround the problem by forcing the import to complete.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

t0yv0 commented 3 months ago

Paraphrasing the import resource option docs the flow is as follows:

programInputs = readFromSource()
ignoreChanges = readIgnoreChangesFromSource()
assumedState, assumedInputs = provider.Read(properties={})
assumedInputsIC = processIgnoreChanges(oldInputs=assumedInputs, newInputs=programInputs)
checkedInputs = provider.Check(assumedInputsIC)
result := provider.Diff(olds=assumedState, oldInputs=assumedInputs, news=checkedInputs, ignoreChanges=ignoreChanges)

if not result.noChanges:
   fail("inputs don't match the program, import will fail")

In my attempt to root cause this issue, the problem arises for bridged providers at as a consequence of a few issues:

they do not apply defaults in Read(), Read() bottoms out at TF ReadResource() API that's not guaranteed to apply defaults
for PF-based resources, Check() does not apply default values, which appears to be assumed to some extend by the engine; it is unable to do so because old state is not available in Check() but is required to run PlanResourceChange where the application happens
ignoreChanges processing happens in three phases, none of which apply here: by the engine prior to Check, by the provider as part of Diff before planning, and by the provider as a post-processing of the planning in difff; the third phase does not apply to PF-based resources and the first two do not apply to this use case.

So when the user does not specify the default value but the provider does, this default value is absent from programInputs, assumedInputs, checkedInputs, and assumedState. At the bottom we get the following diff call:

{
  "method": "/pulumirpc.ResourceProvider/Diff",
  "request": {
    "id": "test-id",
    "urn": "urn:pulumi:test::test::prov:index/test:Test::mainRes",
    "olds": {
      "id": "test-id",
      "otherProp": "val"
    },
    "news": {
      "otherProp": "val"
    },
    "oldInputs": {
      "otherProp": "val"
    }
  },
  "response": {
    "changes": "DIFF_SOME",
    "diffs": [
      "changeReason"
    ]
  }
}

It's suggesting that the provider is going to change the value of changeReason from unset to the provider-initiated default. This happens also if ignoreChanges is applied.

t0yv0 commented 3 months ago

What I find very non-intuitive, besides non-working ignoreChanges, is that if I as a user try to go ahead and edit the source program to match the state provider wants to have by writing out the default value, this still doesn't import:

    properties:
      changeReason: DEFAULT

The technicality here is that changing the program does not fix assumedInputs or assumedState, which still don't have the default value, so no reconciliation is happening.

t0yv0 commented 3 months ago

Sketching out possible solutions here:

we could modify the bridge Read() to try to run PlanResoruceChange so that assumedState, assumedInputs have defaults applied when it is called in the import scenario; it would produce programs with extraneous code to match default values which is not ideal but it would at least get past the Diff check
we could change the protocol so Check receives old state and is able to do PlanResourceChange at that level. Check might need access to ignore_changes as well. This a large and difficult change that turns Check into something like "PlanResourceChange" at Pulumi level. It would fix the problem because Check would then apply defaults.
we could improve Plugin Framework introspection to make sense of Default modifier; this will not be available easily for dynamic or gRPC-based providers, nor would it catch defaults or changes made through PlanModifiers and not the Default option because those defy reflective introspection in principle
we could use this as an argument to introduce ignoreChanges post-processing in Diff for PF resources similar to SDKv2 based resources to try to suppress discovered diffs ex-post-facto; this has some downsides as well

t0yv0 commented 2 months ago

Apologies for the assigned/unassigned noise, to revisit in next iteration with the team and figure out a way forward.

pulumi / pulumi-terraform-bridge