search

IBM-Cloud / terraform-provider-ibm

https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs
Mozilla Public License 2.0
341 stars 666 forks source link

Creating a new topic with an existing event-strem #1977

Closed maximiano86 closed 3 years ago

maximiano86 commented 3 years ago

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

0.12.29

terraform-provider-ibm v1.13.1

Affected Resource(s)

Please list the resources as a list, for example:

Terraform Configuration Files

data "ibm_resource_instance" "ies-kafka-prod-us-south" {
  name = "IES-Kafka-Prod-US"
  resource_group_id = data.ibm_resource_group.engagesupport.id
  service = "messagehub"
}

resource "ibm_event_streams_topic" "ies-kafka-rac-topic-prod-us-south" {
  resource_instance_id = data.ibm_resource_instance.ies-kafka-prod-us-south.id
  name                 = "rac"
  partitions           = 1
  config = {
    "retention.ms"    = "2592000000"
  }
}

Expected Behavior

What should have happened? Create a topic

Actual Behavior

What actually happened? errored out - Error: rpc error: code = Unavailable desc = transport is closing

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply
maximiano86 commented 3 years ago 


2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: 

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: goroutine 161 [running]:

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: github.com/IBM-Cloud/terraform-provider-ibm/ibm.createSaramaAdminClient(0xc00033e690, 0x236af20, 0xc000331900, 0xc000ac56a0, 0xc000aacde0, 0xc000ac56a0, 0x0, 0x0, 0x1fc8d20)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/src/github.com/IBM-Cloud/terraform-provider-ibm/ibm/resource_ibm_event_streams_topic.go:272 +0xea5

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: github.com/IBM-Cloud/terraform-provider-ibm/ibm.resourceIBMEventStreamsTopicCreate(0xc00033e690, 0x236af20, 0xc000331900, 0x2, 0x39a65a0)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/src/github.com/IBM-Cloud/terraform-provider-ibm/ibm/resource_ibm_event_streams_topic.go:134 +0x5a

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).Apply(0xc000848500, 0xc000aae500, 0xc0009e66e0, 0x236af20, 0xc000331900, 0xc0009ec501, 0xc000aacde0, 0xc0009ec658)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.6.0/helper/schema/resource.go:305 +0x365

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Apply(0xc0008b5780, 0xc000ac59c8, 0xc000aae500, 0xc0009e66e0, 0xc000b68bc8, 0xc0003e2510, 0x1fcbea0)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.6.0/helper/schema/provider.go:294 +0x99

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ApplyResourceChange(0xc00000f0a8, 0x296e360, 0xc000aac1b0, 0xc00009ede0, 0xc00000f0a8, 0xc000aac1b0, 0xc000e3db30)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.6.0/internal/helper/plugin/grpc_provider.go:885 +0x8b4

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0x22d7f40, 0xc00000f0a8, 0x296e360, 0xc000aac1b0, 0xc00009ed80, 0x0, 0x296e360, 0xc000aac1b0, 0xc000ab6480, 0x218)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.6.0/internal/tfplugin5/tfplugin5.pb.go:3189 +0x217

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004bd380, 0x2987d80, 0xc000400a80, 0xc0009e8000, 0xc000441aa0, 0x395a0c0, 0x0, 0x0, 0x0)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1024 +0x4f4

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: google.golang.org/grpc.(*Server).handleStream(0xc0004bd380, 0x2987d80, 0xc000400a80, 0xc0009e8000, 0x0)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:1313 +0xd97

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00023e020, 0xc0004bd380, 0x2987d80, 0xc000400a80, 0xc0009e8000)

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:722 +0xbb

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1: created by google.golang.org/grpc.(*Server).serveStreams.func1

2020-10-19T16:54:42.591Z [DEBUG] plugin.terraform-provider-ibm_v1.13.1:     /Users/hkantare/go/pkg/mod/google.golang.org/grpc@v1.27.1/server.go:720 +0xa1

2020-10-19T16:54:42.595Z [DEBUG] plugin: plugin process exited: path=/home/travis/.terraform.d/plugins/terraform-provider-ibm_v1.13.1 pid=5260 error="exit status 2"

2020/10/19 16:54:42 [DEBUG] ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Canceled desc = context canceled

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalMaybeTainted

2020/10/19 16:54:42 [TRACE] EvalMaybeTainted: ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights encountered an error during creation, so it is now marked as tainted

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalWriteState

2020/10/19 16:54:42 [TRACE] EvalWriteState: removing state object for ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalApplyProvisioners

2020/10/19 16:54:42 [TRACE] EvalApplyProvisioners: ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights has no state, so skipping provisioners

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalMaybeTainted

2020-10-19T16:54:42.595Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"

2020/10/19 16:54:42 [TRACE] EvalMaybeTainted: ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights encountered an error during creation, so it is now marked as tainted

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalWriteState

2020/10/19 16:54:42 [TRACE] EvalWriteState: removing state object for ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalIf

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalIf

2020/10/19 16:54:42 [DEBUG] ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalWriteDiff

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalMaybeTainted

2020/10/19 16:54:42 [TRACE] EvalMaybeTainted: ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south encountered an error during creation, so it is now marked as tainted

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalWriteState

2020/10/19 16:54:42 [TRACE] EvalWriteState: removing state object for ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalApplyProvisioners

2020/10/19 16:54:42 [TRACE] EvalApplyProvisioners: ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south has no state, so skipping provisioners

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalMaybeTainted

2020/10/19 16:54:42 [TRACE] EvalMaybeTainted: ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south encountered an error during creation, so it is now marked as tainted

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalWriteState

2020/10/19 16:54:42 [TRACE] EvalWriteState: removing state object for ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalIf

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalIf

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalWriteDiff

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalApplyPost

2020/10/19 16:54:42 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: rpc error: code = Canceled desc = context canceled

2020/10/19 16:54:42 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled

2020/10/19 16:54:42 [TRACE] [walkApply] Exiting eval tree: ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights

2020/10/19 16:54:42 [TRACE] vertex "ibm_iam_access_group_policy.engagesupport_engagesupport-eu-de-prod_service_monitoring-insights": visit complete

2020/10/19 16:54:42 [TRACE] <root>: eval: *terraform.EvalApplyPost

2020/10/19 16:54:42 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing

2020/10/19 16:54:42 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing

2020/10/19 16:54:42 [TRACE] [walkApply] Exiting eval tree: ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south

2020/10/19 16:54:42 [TRACE] vertex "ibm_event_streams_topic.ies-kafka-rac-topic-prod-us-south": visit complete

2020/10/19 16:54:42 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping

2020/10/19 16:54:42 [TRACE] dag/walk: upstream of "provider.ibm (close)" errored, so skipping

2020/10/19 16:54:42 [TRACE] dag/walk: upstream of "root" errored, so skipping

2020-10-19T16:54:42.621Z [DEBUG] plugin: plugin exited

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.

A crash log has been placed at "crash.log" relative to your current

working directory. It would be immensely helpful if you could please

report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That

information is available on the first line of crash.log. You can also

get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 

sensitive information that must be redacted before it is safe to share 

on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!```
JunliWang commented 3 years ago

Could you please provide following information:

  1. the GUID of your Event Streams instance
  2. Does the ibmcloud_api_key have Manager role on the Event Streams instance
maximiano86 commented 3 years ago

1.- GUID: 45e67c6c-60e3-4635-87f6-240118d9ca73

2.- Yes it does, Manager, Writer, Reader

The problem is only when creating a topic in an existing even stream that was not created with terraform (added in a datasource like the example above), if we create a topic in an event stream created with terraform it works just fine.

JunliWang commented 3 years ago

resource_ibm_event_streams_topic.go:272 is adminURL := instance.Extensions["kafka_http_url"].(string) this code is calling Resource Controller API to get this instance and use kafka_http_url to initialize Kafka API client, could you run below commands to see if you can get correct result? ibmcloud login -a cloud.ibm.com --apikey ${ibmcloud_api_key} -g engagesupport -r us-south ibmcloud resource service-instance IES-Kafka-Prod-US --output json

jkayani commented 3 years ago

We do get the correct result logged in with the same API key used for Terraform: 


Joshuas-MBP-2:playbooks jkayani$ bx resource service-instance IES-Kafka-Prod-US --output json
[
    {
        "guid": "1fdee0ba-caf7-433e-adc7-2544c8e5fa65",
        "id": "crn:v1:bluemix:public:messagehub:us-south:a/818a9690e7908c065e8016356406bc5f:1fdee0ba-caf7-433e-adc7-2544c8e5fa65::",
        "url": "/v2/resource_instances/1fdee0ba-caf7-433e-adc7-2544c8e5fa65",
        "created_at": "2020-06-13T16:41:24.33609145Z",
        "updated_at": "2020-06-13T16:41:24.33609145Z",
        "deleted_at": null,
        "name": "IES-Kafka-Prod-US",
        "region_id": "us-south",
        "account_id": "818a9690e7908c065e8016356406bc5f",
        "resource_plan_id": "ibm.eventstreams.standard",
        "resource_group_id": "59fcf11de2c84122ac5ee2d1dece4fe0",
        "crn": "crn:v1:bluemix:public:messagehub:us-south:a/818a9690e7908c065e8016356406bc5f:1fdee0ba-caf7-433e-adc7-2544c8e5fa65::",
        "create_time": 0,
        "created_by": "IBMid-50RAJ86P4C",
        "state": "active",
        "type": "service_instance",
        "resource_id": "6a7f4e38-f218-48ef-9dd2-df408747568e",
        "dashboard_url": "https://mh-prod-control-us-south.us-south.containers.mybluemix.net/index.html",
        "last_operation": {
            "type": "create",
            "state": "succeeded",
            "description": "Completed create instance operation",
            "updated_at": null
        },
        "account_url": "",
        "resource_plan_url": "",
        "resource_bindings_url": "/v2/resource_instances/1fdee0ba-caf7-433e-adc7-2544c8e5fa65/resource_bindings",
        "resource_aliases_url": "/v2/resource_instances/1fdee0ba-caf7-433e-adc7-2544c8e5fa65/resource_aliases",
        "siblings_url": "",
        "target_crn": "crn:v1:bluemix:public:globalcatalog::::deployment:ibm.eventstreams.standard%3Aus-south"
    }
]

I'm a teammate of the person who filed the issue

JunliWang commented 3 years ago

so you provided GUID: 45e67c6c-60e3-4635-87f6-240118d9ca73 but the output from bx resource service-instance IES-Kafka-Prod-US --output json is showing a different instance, CRN is: crn:v1:bluemix:public:messagehub:us-south:a/818a9690e7908c065e8016356406bc5f:1fdee0ba-caf7-433e-adc7-2544c8e5fa65::, its GUID is 1fdee0ba-caf7-433e-adc7-2544c8e5fa65 instance with GUID 45e67c6c-60e3-4635-87f6-240118d9ca73's name should be IES-Kafka-UAT But both instances do not have kafka_http_url in its instance.extensions, those instances were created in June and are not updated to have this kafka_http_url which our terrafrom support is needed. I will fix those instances. Thanks for reporting this problem.

maximiano86 commented 3 years ago

can you do the same for IES-Kafka-Prod-EU as well, thanks in advance

JunliWang commented 3 years ago

both IES-Kafka-UAT and IES-Kafka-Prod-US are manually updated, code fix will come later. Could you give me the CRN of instance IES-Kafka-Prod-EU?

maximiano86 commented 3 years ago 

[
    {
        "guid": "3e122365-697b-4c3d-8a8a-423580d03da9",
        "id": "crn:v1:bluemix:public:messagehub:eu-de:a/818a9690e7908c065e8016356406bc5f:3e122365-697b-4c3d-8a8a-423580d03da9::",
        "url": "/v2/resource_instances/3e122365-697b-4c3d-8a8a-423580d03da9",
        "created_at": "2020-06-13T21:59:21.321823061Z",
        "updated_at": "2020-06-13T21:59:21.321823061Z",
        "deleted_at": null,
        "name": "IES-Kafka-Prod-EU",
        "region_id": "eu-de",
        "account_id": "818a9690e7908c065e8016356406bc5f",
        "resource_plan_id": "ibm.eventstreams.standard",
        "resource_group_id": "59fcf11de2c84122ac5ee2d1dece4fe0",
        "crn": "crn:v1:bluemix:public:messagehub:eu-de:a/818a9690e7908c065e8016356406bc5f:3e122365-697b-4c3d-8a8a-423580d03da9::",
        "create_time": 0,
        "created_by": "IBMid-50RAJ86P4C",
        "state": "active",
        "type": "service_instance",
        "resource_id": "6a7f4e38-f218-48ef-9dd2-df408747568e",
        "dashboard_url": "https://mh-prod-control-e-282592.eu-de.containers.appdomain.cloud/index.html",
        "last_operation": {
            "type": "create",
            "state": "succeeded",
            "description": "Completed create instance operation",
            "updated_at": null
        },
        "account_url": "",
        "resource_plan_url": "",
        "resource_bindings_url": "/v2/resource_instances/3e122365-697b-4c3d-8a8a-423580d03da9/resource_bindings",
        "resource_aliases_url": "/v2/resource_instances/3e122365-697b-4c3d-8a8a-423580d03da9/resource_aliases",
        "siblings_url": "",
        "target_crn": "crn:v1:bluemix:public:globalcatalog::::deployment:ibm.eventstreams.standard%3Aeu-de"
    }
]
JunliWang commented 3 years ago

updated this one too.

JunliWang commented 3 years ago

the fix is deployed to prod, this issue could be closed.