specmesh / specmesh-build

https://specmesh.io/
Apache License 2.0
13 stars 5 forks source link

Support key schemas and types #319

Closed big-andy-coates closed 4 months ago

big-andy-coates commented 4 months ago

fixes: #301 replaces: https://github.com/specmesh/specmesh-build/pull/318 & https://github.com/specmesh/specmesh-build/pull/304

BREAKING CHANGE: The API of the SchemaInfo type has changed, which can cause compilation issues.

With Kafka, different parts of the message payload, i.e. the Kafka record, can have type and schema information associated with them. Most people think of the record value, but there is also the key to think about, and in the future the headers.

This change seeings support for providing type & schema information for the key of each Kafka record in a topic. This is done by extracting the key information from the channel's message.bindings.kafka.key field.

For example, the following defines a Kafka topic with a key that contains a complex type:

  message:
    bindings:
      kafka:
        key:
          $ref: "/schema/key.avsc"
    payload:
      $ref: "/schema/value.avsc"

The key, or indeed the value, can also be defined as holding one of the inbuilt Kafka types, i.e. types for which most Kafka clients provide serializers out of the box, e.g.

  message:
    bindings:
      kafka:
        key:
          type: string
    payload:
      $ref: "/schema/value.avsc"

Support for inline schemas for both key and value remains, however, the provisioning of inline schemas is still unsupported.

The Provisioner api remains backwards compatible.

The SchemaInfo class has changed, with optional fields now encoded as Optional types.

KafkaApiSpec.schemaInfoForTopic is now marked as deprecated, as a topic can now define multiple schema. Use ownedTopicSchemas as a direct replacement, or topicSchemas to get all schemas associated with a topic, rather than just those required for provisioning.