Schema references

[noureddineseddik]

Why do we need this

see #13

I tried to respect the existing codebase: reuse the same test cases, variable naming, ... This PR is pretty big, but it's mostly tests and documentation, could not find a way to split it. I can rework it if you think of something.

Given the schema a is updated, the schema b version should be updated and its reference to schema a version also. If one wants to stick with a version he can just hardcode it in the reference block

Manually sticking a version is actually harder than expected and is not covered (more on this below). In that PR, I propose to always update a reference version to the latest referenced schema's one.

What changed

• DataSource with schema references support & tests:

  • references exposed as output value
  • In the reference test setup, create in test setup a referenced schema and a schema with reference using the registry client
  • Update riferrei/srclient dependency from 0.3.0 to 0.3.1 to fetch references from schema registry

• Resource with schema references support & tests:

  • CRU operations handling references
  • Introduce CustomizeDiff schema function to explicitly set a version change on schema change and make dependencies aware of a version changed at plan time (computed field)
  • Fixture text templates as schemas with references are complex to handle with simple string format

• Examples and readme updated:

  • Update the main.tf required provider version to 0.6

Sticking a reference to a specific version

This is my first terraform dev, but as I understand it, referencing a referenced resource schema in a given version while updating this referenced schema to a new version would require duplicating the resource, something like:

resource "schemaregistry_schema" "user_added_latest" {
  subject = "MyTopic-akc.test.userAdded-value"
  schema  = file("./userAdded_v2.avsc")
}

resource "schemaregistry_schema" "user_added_v1" {
  subject = "MyTopic-akc.test.userAdded-value"
  schema  = file("./userAdded.avsc")
}

resource "schemaregistry_schema" "with_reference" {
  subject = "with-reference"
  schema = "[\"akc.test.userAdded\"]"
  references {
    name = "akc.test.userAdded"
    subject = schemaregistry_schema.user_added_v1.subject
    version = 1
  }
} 

This is not possible with current implementation as the schema subject is used as a terraform schema ID. One way I see could be to rely on schema id instead but that would introduce too many changes for that PR.

Notes

• locally tested terraform plan & apply
• relied on the docker containers:
  • broker: confluentinc/cp-server:6.1.1
  • schema-registry: confluentinc/cp-schema-registry:6.1.1
  • zookeeper: confluentinc/cp-zookeeper:6.1.1

[arkiaconsulting]

Thanks @noureddineseddik,

when I started to implement this provider, I considered using the schema Id as the terraform Id. But the problem is that a schema Id is unique across the schema registry, but it can be used onto several subjects. For example, if you create a subject with a given schema content, and another subject with the same schema content, those 2 schemas will get the same schema Id. so you would end up with 2 subjects having the same terraform Id => fail

Then I considered separating the subjects and the schemas, having 2 terraform resources. But the SR Api does not offer a way to create a schema without a subject.

To be honest, given the SR Api current state, I don't see a way to handle this properly...

Do you have anything to add ?

[noureddineseddik]

Thanks, I missed that ID part. My proposition of having 2 terraform resources to handle the same subject does not make sense anyway.

Is it acceptable to always update a reference version to the latest referenced schema's one as currently implemented in that PR?

[dstendardi]

Nice contribution @noureddineseddik ! @arkiaconsulting we would be very interested to have this feature merged 💯

[arkiaconsulting]

Here we go