search

pulumi / pulumi-confluentcloud

A Confluent Pulumi resource package, providing multi-language access to Confluent
Apache License 2.0
8 stars 3 forks source link

panic: interface conversion: interface {} is nil, not string pulumi cli 3.86.0 #234

Closed tusharshahrs closed 1 year ago

tusharshahrs commented 1 year ago

What happened?

We are trying to manage with pulumi our kafka clusters on Confluent Cloud and as part of the cluster creation i'm trying to create a Cluster API key with pulumi. However i'm getting the following unexpected weird error:

View Live: https://app.pulumi.com/.....e2e_kafka/e2e-kafka/updates/6

Type Name Status Info
Type Name Status Info pulumi:pulumi:Stack e2e_kafka-e2e-kafka **failed** 1 error; 29 mes
└─ KafkaCreate kurrents_create_template_instance
└─ pulumi:providers:confluentcloud kafka_provider_kurrents
└─ confluentcloud:index:KafkaCluster kafka_cluster_kurrents-e2e
└─ confluentcloud:index:ApiKey kafka_cluster_kurrents-e2e-credentials
+ └─ confluentcloud:index:KafkaAcl kafka_cluster_kurrents-e2e-credentials_acl_rule **creating failed** 1 error

Diagnostics:
pulumi:pulumi:Stack (e2e_kafka-e2e-kafka):
2023/10/05 16:51:17 [DEBUG] GET https://api.confluent.cloud/iam/v2/service-accounts/sa-rrrrg7
panic: interface conversion: interface {} is nil, not string
goroutine 11 [running]:
github.com/confluentinc/terraform-provider-confluent/internal/provider.extractRestEndpoint(0x30?, 0x2281240?, 0x1?)
/home/runner/work/pulumi-confluentcloud/pulumi-confluentcloud/upstream/internal/provider/resource_kafka_topic.go:178 +0x170
github.com/confluentinc/terraform-provider-confluent/internal/provider.kafkaAclCreate({0x258c110, 0xc000418570}, 0xc0002eb900, {0x2277020?, 0xc0005f9180?})
/home/runner/work/pulumi-confluentcloud/pulumi-confluentcloud/upstream/internal/provider/resource_kafka_acl.go:150 +0x8e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc00001ac40, {0x258c0d8, 0xc00012c010}, 0xd?, {0x2277020, 0xc0005f9180})
/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[v2@v2.0.0-20230710100801-03a71d0fca3d](mailto:v2@v2.0.0-20230710100801-03a71d0fca3d)/helper/schema/resource.go:738 +0x12e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc00001ac40, {0x258c0d8, 0xc00012c010}, 0x0, 0xc0002eb700, {0x2277020, 0xc0005f9180})
/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[v2@v2.0.0-20230710100801-03a71d0fca3d](mailto:v2@v2.0.0-20230710100801-03a71d0fca3d)/helper/schema/resource.go:869 +0xa7d
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0xc000788060?, {0x0?, 0xc0002eb700?, 0x0?}}, {0x231082e, 0x13}, {0x0?, 0x0}, {0x25929d0, 0xc0002eb700})
/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[v3@v3.55.0](mailto:v3@v3.55.0)/pkg/tfshim/sdk-v2/provider.go:100 +0x188
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Create(0xc0002a6580, {0x258c148?, 0xc0003f2f60?}, 0xc0003626e0)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[v3@v3.55.0](mailto:v3@v3.55.0)/pkg/tfbridge/provider.go:743 +0x663
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Create_Handler.func1({0x258c148, 0xc0003f2f60}, {0x2223f80?, 0xc0003626e0})
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[v3@v3.76.1](mailto:v3@v3.76.1)/proto/go/provider_grpc.pb.go:573 +0x78
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x258c148, 0xc0003f2b70}, {0x2223f80, 0xc0003626e0}, 0xc0008be440, 0xc000125518)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645](mailto:grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645)/go/otgrpc/server.go:57 +0x3e8
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Create_Handler({0x22bf2c0?, 0xc0002a6580}, {0x258c148, 0xc0003f2b70}, 0xc00035e1c0, 0xc0003f4460)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[v3@v3.76.1](mailto:v3@v3.76.1)/proto/go/provider_grpc.pb.go:575 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000564000, {0x2592da0, 0xc00045e000}, 0xc0008bab40, 0xc0000b9050, 0x30d6600, 0x0)
/home/runner/go/pkg/mod/google.golang.org/[grpc@v1.56.1](mailto:grpc@v1.56.1)/server.go:1337 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc000564000, {0x2592da0, 0xc00045e000}, 0xc0008bab40, 0x0)
/home/runner/go/pkg/mod/google.golang.org/[grpc@v1.56.1](mailto:grpc@v1.56.1)/server.go:1714 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
/home/runner/go/pkg/mod/google.golang.org/[grpc@v1.56.1](mailto:grpc@v1.56.1)/server.go:959 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/runner/go/pkg/mod/google.golang.org/[grpc@v1.56.1](mailto:grpc@v1.56.1)/server.go:957 +0x18c

error: update failed

confluentcloud:index:KafkaAcl (kafka_cluster_kurrents-e2e-credentials_acl_rule):
error: error reading from server: EOF

It's important to note that i'm getting it only when i run pulumi-up and not in pulumi-preview. I've attached here files with the source code i'm using to create the cluster, api_key and the cluster_acl_rule.

Example

Here is the code

from pulumi import ComponentResource, ResourceOptions, Output, InvokeOptions
from typing import Union, Optional, List, Dict, Type
from .constants import VAULT_ADMIN
import pulumi_confluentcloud as ccloud
from pulumi_komponents.common.vault_config import VaultClientOperations
import pulumi_vault

class Kafka(ComponentResource):

    def __init__(self, args):
        self.args = args
        self.project_name = args.project_name
        self.environment = args.environment
        self.region = args.region
        self.vault_environment = args.vault_environment
        super().__init__('KafkaCreate', f"{self.project_name}_create_template_instance")
        super().register_outputs({})

        self.confluent_env_id = self.admin_credentials.get(VAULT_ADMIN["confluent_env_key"])
        self.confluent_provider = self._ccloud_provider()
        self.created_kafka_cluster = self.create_kafka_cluster()
        self.pulumi_service_account = self._get_service_account()
        self.created_cluster_apikey = self.create_cluster_apikey()
        # self.created_vault_secret_apikey = self._create_vault_secret_apikey()

    @property
    def vault_client(self) -> VaultClientOperations:
        try:
            return self._vault_client
        except AttributeError:
            self._vault_client = VaultClientOperations(self.vault_environment)
            return self._vault_client

    @property
    def admin_credentials(self) -> Dict[str, str]:
        '''
        This property will send request to Vault and return secret in dictionary type
        '''
        return self.vault_client.get_secret_value_from_vault(VAULT_ADMIN["path"])

    def _ccloud_provider(self):
        return ccloud.Provider(f"kafka_provider_{self.project_name}",
                               cloud_api_key=self.admin_credentials.get(VAULT_ADMIN["api_key"]),
                               cloud_api_secret=self.admin_credentials.get(VAULT_ADMIN["api_secret_key"]),
                               opts=ResourceOptions(parent=self))

    def create_kafka_cluster(self):
        cluster_name = f"kafka_cluster_{self.project_name}-{self.environment}"
        kafka_cluster_args = ccloud.KafkaClusterArgs(
                                        display_name=f"{self.project_name}-{self.environment}",
                                        cloud=self.args.cloud_provider,
                                        availability=self.args.cluster_availability,
                                        region=self.region,
                                        environment=ccloud.KafkaClusterEnvironmentArgs(id=self.confluent_env_id),
                                        dedicated=ccloud.KafkaClusterDedicatedArgs(cku=int(self.args.cluster_cku)))

        return ccloud.KafkaCluster(resource_name=cluster_name,
                                   args=kafka_cluster_args,
                                   opts=ResourceOptions(provider=self.confluent_provider,
                                                        parent=self.confluent_provider,
                                                        depends_on=self.confluent_provider))
    def _get_service_account(self):
        return ccloud.get_service_account(id="sa-rrrrg7",
                                          opts=InvokeOptions(provider=self.confluent_provider,
                                                             parent=self.confluent_provider))

    def create_cluster_apikey(self):
        resource_name = f"kafka_cluster_{self.project_name}-{self.environment}-credentials"
        try:
            owner_args = ccloud.ApiKeyOwnerArgs(api_version=self.pulumi_service_account.api_version,
                                                id=self.pulumi_service_account.id,
                                                kind=self.pulumi_service_account.kind)

            managed_resource_args = ccloud.ApiKeyManagedResourceArgs(id=self.created_kafka_cluster.id,
                                                                    api_version="cmk/v2",
                                                                    environment=ccloud.ApiKeyManagedResourceEnvironmentArgs(id=self.confluent_env_id),
                                                                    kind="Cluster")

            api_key = ccloud.ApiKey(resource_name,
                                display_name=resource_name,
                                description=f"initial {self.project_name}-{self.environment} api key",
                                owner=owner_args,
                                managed_resource=managed_resource_args,
                                opts=ResourceOptions(provider=self.confluent_provider,
                                                    parent=self.created_kafka_cluster,
                                                    depends_on=self.created_kafka_cluster))

            acl_rule = ccloud.KafkaAcl(f"{resource_name}_acl_rule",
                                kafka_cluster=ccloud.KafkaAclKafkaClusterArgs(id=self.created_kafka_cluster.id),
                                operation="ALL",
                                permission="ALLOW",
                                host="*",
                                pattern_type="LITERAL",
                                principal=f"User:{self.pulumi_service_account.id}",
                                resource_type="TOPIC", 
                                resource_name_="*",
                                opts=ResourceOptions(provider=self.confluent_provider,
                                                        parent=api_key,
                                                        depends_on=[api_key]))

            return api_key
        except Exception as e:
            # pulumi.log.error(f"Error creating API key and ACL rule: {str(e)}")
            print(str(e))

    def _create_vault_secret_apikey(self) -> pulumi_vault.generic.Secret:
        secret_resource_name = f"{self.project_name}-{self.environment}_apikey_credentials"
        secret_json = Output.json_dumps({
            'cluster_apiaccess_key': self.created_cluster_apikey.id,
            'cluster_apiaccess_secret': self.created_cluster_apikey.secret
        })

        created_secret = pulumi_vault.generic.Secret(resource_name=secret_resource_name,
                                                     path=f"{VAULT_ADMIN['path']}/{self.project_name}-{self.environment}",
                                                     data_json=secret_json,
                                                     opts=ResourceOptions(provider=self.vault_client.provider,
                                                                          parent=self.vault_client.provider,
                                                                          depends_on=self.created_cluster_apikey))
        return created_secret

Issue only happens on pulumi-up and not in pulumi-preview.

Output of pulumi about

pulumi about
CLI
Version 3.51.1
Go Version go1.19.5
Go Compiler gc Host
OS darwin
Version 12.4
Arch x86_64 Backend
Name pulumi.com
URL https://app.pulumi.com/....username......


requested

Additional context

requirements.txt

pulumi-komponents.common
pulumi==3.86.0
pulumi-aws==4.*
protobuf==4.21
pulumi_confluentcloud

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

mikhailshilkov commented 1 year ago

Hi @tusharshahrs what is the version of pulumi_confluentcloud you are on? I assume the latest, but somehow stack trace lines don't match up, so I want to double-check.

The panic comes from the upstream provider. Looking at the code, it sounds like it's a poor handling of an error that should manifest as the following error related to ccloud.KafkaAcl:

one of provider.kafka_rest_endpoint (defaults to KAFKA_REST_ENDPOINT environment variable) or resource.rest_endpoint must be set

I don't see this option set in your code, could you give it a try?

tusharshahrs commented 1 year ago

They are on version: pulumi-confluentcloud, Version:1.24.0

tusharshahrs commented 1 year ago

Requested the customer to set the option for the KAFKA_REST_ENDPOINT. Waiting to hear back.

tusharshahrs commented 1 year ago 
[Yuda Pinhas](https://pulumi.zendesk.com/agent/users/13609993408403)
57 minutes ago
I have done the suggestion to add the kafka_rest_end. It's working!

Closing the panic issue.

pulumi-bot commented 1 year ago

Cannot close issue:

Please fix these problems and try again.