Creation of Schema Registry fails with 400: Failed to Decode Request

[aviflax]

I’m trying to provision a Schema Registry in a Confluent Cloud Environment, using this provider, and it’s not working for me.

Steps to reproduce

My resource definition looks like this:

this.schema_registry = new confluent.SchemaRegistry(name, {
  environmentId: this.environment.id,
  region: aws_config.require("region"),
  serviceProvider: "AWS",
}, {
  parent: this,
  // Requires at least one kafka cluster to enable the schema registry in the environment.
  dependsOn: this.cluster,
});

…which I believe is more or less correct, and I expected this to result in a Schema Registry being successfully provisioned in my Confluent Cloud Environment.

Instead, I get this error:

  ==============================================================================
  2021/07/12 14:00:12.592347 DEBUG RESTY
  ==============================================================================
  ~~~ REQUEST ~~~
  POST  /api/schema_registries?account_id=env-6n2yj  HTTP/1.1
  HOST   : confluent.cloud
  HEADERS:
    Content-Type: application/json
    User-Agent: go-confluent-cloud 0.1
  BODY   :
  {
      "config": {
        "account_id": "env-6n2yj",
        "kafka_cluster_id": "",
        "location": "eu-central-1",
        "name": "account schema-registry",
        "service_provider": "AWS"
      }
  }
  ------------------------------------------------------------------------------
  ~~~ RESPONSE ~~~
  STATUS       : 400 Bad Request
  RECEIVED AT  : 2021-07-12T14:00:12.592261-04:00
  TIME DURATION: 124.079375ms
  HEADERS      :
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Headers: Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range
    Access-Control-Allow-Methods: GET,POST,OPTIONS,PUT,DELETE,PATCH
    Connection: keep-alive
    Content-Length: 60
    Content-Type: application/json; charset=utf-8
    Date: Mon, 12 Jul 2021 18:00:12 GMT
    Server: nginx
    Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
    X-Content-Type-Options: nosniff
    X-Frame-Options: deny
    X-Request-Id: 83c44e20d7cf27fb0d2c46df0ca5c26d
    X-Xss-Protection: 1; mode=block
  BODY         :
  {
      "error": {
        "code": 400,
        "message": "Failed to Decode Request"
      }
  }
  ==============================================================================

confluent:index:SchemaRegistry (main-dev-bilal):
  error: 1 error occurred:
    * Schema registry: Failed to Decode Request

I suspect that the problem is related to the kv pair "kafka_cluster_id": "" in the request body, but I can’t say for sure.

For now I’ve worked around this with a dynamic provider that invokes the ccloud CLI tool, but it’s a ton of code that I’d love to remove from my project at some point.

Thank you!

[emiliza]

@aviflax Underlying CreateSchemaRegistry from ccloud actually produces the ”” for Kafka cluster id: https://github.com/cgroschupp/go-client-confluent-cloud/blob/98176441a5a5e49dca8f893bd63c8e9f08be09fd/confluentcloud/schema_registry.go#L109, so this might be ok

Can't find any documentation but looking at some examples it looks like they set the service provider to be all lowercased. Can you try using serviceProvider: "aws" and see if that perhaps solves your problem?

[aviflax]

Hi @emiliza thank you for taking a look, and thanks for the suggestion!

While perhaps that change is indeed necessary, unfortunately it doesn’t seem to be sufficient; the request is still failing:

Failing request and error response

```text ============================================================================== 2021/07/14 12:33:11.937504 DEBUG RESTY ============================================================================== ~~~ REQUEST ~~~ POST /api/schema_registries?account_id=env-0k0w6 HTTP/1.1 HOST : confluent.cloud HEADERS: Content-Type: application/json User-Agent: go-confluent-cloud 0.1 BODY : { "config": { "account_id": "env-0k0w6", "kafka_cluster_id": "", "location": "eu-central-1", "name": "account schema-registry", "service_provider": "aws" } } ------------------------------------------------------------------------------ ~~~ RESPONSE ~~~ STATUS : 400 Bad Request RECEIVED AT : 2021-07-14T12:33:11.937316-04:00 TIME DURATION: 118.344167ms HEADERS : Access-Control-Allow-Credentials: true Access-Control-Allow-Headers: Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range Access-Control-Allow-Methods: GET,POST,OPTIONS,PUT,DELETE,PATCH Connection: keep-alive Content-Length: 60 Content-Type: application/json; charset=utf-8 Date: Wed, 14 Jul 2021 16:33:11 GMT Server: nginx Strict-Transport-Security: max-age=31536000; includeSubDomains; preload X-Content-Type-Options: nosniff X-Frame-Options: deny X-Request-Id: d1c18f6a8f37634a200d47d3ddcd66aa X-Xss-Protection: 1; mode=block BODY : { "error": { "code": 400, "message": "Failed to Decode Request" } } ```

I’m frustrated that the Confluent Cloud API returns such vague error messages.

If you have any other suggestions, I’d greatly appreciate any further help — thank you!

[aviflax]

So, I thought I’d take a closer look at the source code for the Terraform provider that this provider builds on, and look again to see if there are any tests. Last time I looked I didn’t see any tests, but I thought I’d check again and look more closely.

Turns out that while there are very few (only one) .go files with “test” in their name, there is a test script that runs the example Terraform module. I think that’s why I missed it the first time I looked — because it didn’t occur to me that the example might actually also be a test. (I do think it’s actually quite smart; it’s a way of ensuring that the documentation stays up to date.)

Anyway, I took a closer look at the example declaration of the resource confluentcloud_schema_registry and thought I’d look for differences between the test/example values used in that file and the values my code was using — as inspired by your suggestion that there might be a problem with the property service_provider. And I found one! Turns out my problem was specifying the value of location as eu-central-1 (the ID of the AWS region my cluster is in) — apparently the Confluent Cloud API wants the value of this property to be, in my case, EU — once I switched it, that worked fine.

This is a little surprising (in a bad way) because the request to create a Kafka cluster also has a property named region and in that case, the value eu-central-1 works just fine. But I guess that’s something to take up with the maintainers of the API, not with y’all.

So: I’m good! Thanks for the help!

---

I’m going to leave this open for now, just in case there’s any further follow-up action y’all want to take related to my experience (e.g. maybe contribute better error messages and/or property validation/guidance into the upstream provider) — but feel free to close this whenever y’all want.

Thank you!

[emiliza]

Glad this works for you! I'll just mark this as resolved since we're not actively tracking a fix upstream.