search

pulumi / pulumi-kafka

A Kafka Pulumi resource package, providing multi-language access to Kafka
Apache License 2.0
16 stars 1 forks source link

Potential bug for User Import #208

Closed seed-doordash closed 1 year ago

seed-doordash commented 1 year ago

What happened?

We are trying to migrate from TF to Pulumi. When importing a Kafka Scram User, we cannot get this to work.

Our program code roughly looks like this (some parts ommitted)

    // handle import
    opts := make([]pulumi.ResourceOption, 0)
    if config.GetBool(ctx, "is_import") {
            resourceId := config.Get(ctx, "resource_import_id")
        opts = append(opts, pulumi.Import(pulumi.ID(resourceId)))
    }
    options := append(opts, pulumi.Provider(kafkaProvider))
    if password == (pulumi.StringOutput{}) {
        return nil, errors.New("password not set")
    }

    args.UserScramCredentialArgs.Password = password

    user, err := kafka.NewUserScramCredential(
        ctx,
        fmt.Sprintf("%s", args.Username),
        args.UserScramCredentialArgs,
        options...,
    )
    if err != nil {
        return nil, err
    }
    ctx.Export("id", user.ID())
    ctx.Export("updatedAt", pulumi.StringPtr(fmt.Sprintf("%d", time.Now().Unix())))
    ctx.Export("password", password)

    return user, nil

Our config looks like

encryptionsalt: xxxxxx
config:
  username: "self_server_import_test_user_1"
  project: "storage-infra"
  owning_team: "TF-SG - PE Storage"
  max_produce_bytes: "-1"
  max_consume_bytes: "-1"
  cluster: "dd-kafka-local"
  target: "local"
  type: "kafka-user"
  name: "self_server_import_test_user_1"
  is_import: "true"
  vault_mode: "EXISTS"
  resource_import_id: "self_server_import_test_user_1|SCRAM-SHA-256"

The pulumi preview shows

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing update (user-test-import):
     Type                                Name                                           Plan       Info
 +   pulumi:pulumi:Stack                 kafka-user-user-test-import                    create     1 error
 +   ├─ pulumi:providers:vault           vaultProvider                                  create
 +   ├─ pulumi:providers:kafka           kafka                                          create
 =   ├─ kafka:index:UserScramCredential  self_server_import_test_user_1                 import     1 error
 =   └─ kafka:index:Acl                  self_server_import_test_user_1-prefixed-group  import
Diagnostics:
  kafka:index:UserScramCredential (self_server_import_test_user_1):
    error: Preview failed: resource 'self_server_import_test_user_1|SCRAM-SHA-256' does not exist

  pulumi:pulumi:Stack (kafka-user-user-test-import):
    error: preview failed

However, the resource def exits, i even checked the TF state, it looks like this

 {
      "module": "module.self_server_import_test_user_1",
      "mode": "managed",
      "type": "kafka_user_scram_credential",
      "name": "kafka_user",
      "provider": "provider[\"registry.terraform.io/mongey/kafka\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "self_server_import_test_user_1|SCRAM-SHA-256",
            "password": "REDACTED",
            "scram_iterations": 4096,
            "scram_mechanism": "SCRAM-SHA-256",
            "username": "self_server_import_test_user_1"
          },
          "sensitive_attributes": [
            [
              {
                "type": "get_attr",
                "value": "password"
              }
            ]
          ],
          "private": "bnVsbA==",
          "dependencies": [
            "data.aws_secretsmanager_secret_version.password",
            "data.aws_secretsmanager_secret_version.username",
            "module.self_server_import_test_user_1.aws_secretsmanager_secret.kafka_user_password",
            "module.self_server_import_test_user_1.aws_secretsmanager_secret_version.kafka_user_password",
            "module.self_server_import_test_user_1.random_password.password"
          ]
        }
      ]
    },

So the ID is def self_server_import_test_user_1|SCRAM-SHA-256, what's going on here?

Expected Behavior

should be able to import

Steps to reproduce

  1. pulumi login
  2. pulumi init
  3. edit the config to be 
    config:
username: "self_server_import_test_user_1"
project: "storage-infra"
owning_team: "TF-SG - PE Storage"
max_produce_bytes: "-1"
max_consume_bytes: "-1"
cluster: "dd-kafka-local"
target: "local"
type: "kafka-user"
name: "self_server_import_test_user_1"
is_import: "true"
vault_mode: "EXISTS"
resource_import_id: "self_server_import_test_user_1|SCRAM-SHA-256"
  4. run pulumi preview

Output of pulumi about

❯ pulumi about
CLI
Version      3.82.0
Go Version   go1.21.1
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.17.0
go      unknown
kafka   3.5.0
random  4.13.0
vault   5.13.0
vault   4.6.0

Host
OS       darwin
Version  13.5
Arch     arm64

This project is written in go: executable='/usr/local/go/bin/go' version='go version go1.21.1 darwin/arm64'

Backend
Name           YK39G2JQXG
URL            s3://doordash-pulumi-state-local
User           seedzeng
Organizations

Dependencies:
NAME                                    VERSION
github.com/go-playground/validator/v10  10.11.2
github.com/hashicorp/go-multierror      1.1.1
github.com/pulumi/pulumi-aws/sdk/v5     5.17.0
github.com/pulumi/pulumi-kafka/sdk/v3   3.5.0
github.com/pulumi/pulumi-random/sdk/v4  4.13.0
github.com/pulumi/pulumi-vault/sdk/v4   4.6.0
github.com/pulumi/pulumi-vault/sdk/v5   5.13.0
github.com/pulumi/pulumi/sdk/v3         3.76.0
github.com/rs/zerolog                   1.28.0
github.com/stretchr/testify             1.8.3
github.com/vmanikes/Functionality       0.1.0
k8s.io/utils                            0.0.0-20221128185143-99ec85e7a44

Additional context

We are able to import all other resources like topic and acl. It's just the scram user we are having trouble.

The "id" for other resources in the TF state all matchs

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

mikhailshilkov commented 1 year ago

Hi @seed-doordash

I'm curious - would the import command work if you run it from the CLI? Something like 

pulumi import kafka:index/userScramCredential:UserScramCredential test "self_server_import_test_user_1|SCRAM-SHA256"
seed-doordash commented 1 year ago

@mikhailshilkov thanks for response. I get

 +   pulumi:pulumi:Stack  kafka-user-user-test-import  create
panic: fatal: An assertion has failed: inputs was non-nil

                                                         goroutine 101 [running]:
 github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
                                                                        /private/tmp/pulumi-20230912-4658-1x1bwwe/sdk/go/common/util/contract/failfast.go:23
                                                                            github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0xc0?, {0x103f79b1d?, 0x10475a140?}, {0x0?, 0x10475a140?, 0x105db14a8?})
                                                        /private/tmp/pulumi-20230912-4658-1x1bwwe/sdk/go/common/util/contract/assert.go:35 +0xe0
                                                                github.com/pulumi/pulumi/sdk/v3/go/common/resource.NewState({0x1400005b488, 0x16}, {0x140010a9cc0, 0x4e}, 0x1, 0x0, {0x0, _}, _, _, ...)
                                            /private/tmp/pulumi-20230912-4658-1x1bwwe/sdk/go/common/resource/resource_state.go:79 +0xe4
                                                           github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*importer).registerProviders(0x140014ddb00, {0x104c9c6f0, 0x140013d8280})
                        /private/tmp/pulumi-20230912-4658-1x1bwwe/pkg/resource/deploy/import.go:261 +0xc24
                          github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*importer).importResources(0x140014ddb00, {0x104c9c6f0, 0x140013d8280})
                                                                    /private/tmp/pulumi-20230912-4658-1x1bwwe/pkg/resource/deploy/import.go:303 +0xa0
                                                                         github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).importResources(0x14001226ba0, {0x104c9c6b8, 0x140013ca780}, {{0x14f69e898, 0x140004de000}, 0x7fffffff, 0x0, 0x0, {{0x0, 0x0, ...}, ...}, ...}, ...)
                                                    /private/tmp/pulumi-20230912-4658-1x1bwwe/pkg/resource/deploy/deployment_executor.go:435 +0xc8
                                                                      github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute(0x14001226ba0, {0x104c9c6b8?, 0x140013ca780}, {{0x14f69e898, 0x140004de000}, 0x7fffffff, 0x0, 0x0, {{0x0, 0x0, ...}, ...}, ...}, ...)
                                            /private/tmp/pulumi-20230912-4658-1x1bwwe/pkg/resource/deploy/deployment_executor.go:144 +0x750
                                                               github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*Deployment).Execute(...)
                                                            /private/tmp/pulumi-20230912-4658-1x1bwwe/pkg/resource/deploy/deployment.go:565
                                                               github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run.func1()
                                                /private/tmp/pulumi-20230912-4658-1x1bwwe/pkg/engine/deployment.go:301 +0x1b4
                                             created by github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run in goroutine 1
                                                /private/tmp/pulumi-20230912-4658-1x1bwwe/pkg/engine/deployment.go:286 +0x1fc
                                             %
░▒▓ ~/Projects/pulumi-projects/go/kafka/user  kanedu/user-vault-import !2 ▓▒░
seed-doordash commented 1 year ago

It seems like it's the underlying TF provider does not support import. Our walkaround is gonna be a "overriding import" (recreate)

gonna close out this issue