TaskMetadata as header

[ocadaruma]

Rework of https://github.com/line/decaton/pull/80 since it's too outdated.

Motivation

• Currently, DecatonClient serializes tasks in DecatonTaskRequest protobuf format because when Decaton had started, Kafka didn't have record header yet
  • As Kafka started record header support quite long ago, it's natural to use it to embed task metadata

Summary of changes

Deprecate DecatonTaskRequest

• DecatonTaskRequest protobuf is necessary only to parse decaton tasks produced from old clients/retry-processors), so I moved it to .internal and marked as deprecated
• DecatonTaskRequest will be removed in later major release in the future

  DecatonClient

• DecatonClient now produces serialized tasks as record value directly, instead of wrapping as DecatonTaskRequest
• TaskMetadata is stored in record headers from now on

  TaskExtractor#extract signature change

• To extract TaskMetadata from record headers, TaskExtractor#extract now accepts ConsumerRecord<byte[], byte[]> instead of just byte[]

  ProcessorProperties, RetryQueueingProcessor

• New decaton.task.metadata.as.header config is introduced. This is to control producing retry tasks with header-metadata or in deprecated DecatonTaskRequest format.

  Trivial

• Updated copyright header config

Breaking changes

• DecatonTaskRequest package change to com.linecorp.decaton.protocol.internal
• KafkaProducerSupplier signature change
• DecatonClient no longer wraps tasks as DecatonTaskRequest
• TaskExtractor signature change
• Retry tasks are no longer wrapped as DecatonTaskRequest unless decaton.task.metadata.as.header is set to false

IMPORTANT NOTE

• To upgrade Decaton to 9.0.0 (which will contains this PR) from prior releases, users MUST take care about the procedure, because topic format is changed. Otherwise the processor may cause error.
• General rule:
  • You must upgrade processors first, before upgrading clients
• Procedures depending on the use case:
  • A: Using retry-queueing && downtime is not acceptable:
    • (1) Upgrade all processors with setting decaton.task.metadata.as.header = false
    • (2) Upgrade clients
    • (3) Set decaton.task.metadata.as.header = true on your timing later
  • B: Using retry-queueing && downtime is acceptable:
    • (1) Shutdown all processors once, then start processors with new version
    • (2) Upgrade clients
  • C: Not using retry-queueing:
    • (1) Upgrade processors as usual
    • (2) Upgrade clients

TODOs

• Prepare a detailed migration guide and include it in release note