search

segmentio / topicctl

Tool for declarative management of Kafka topics
MIT License
590 stars 55 forks source link

Unexpected drift reported for message.timestamp.before.max.ms, message.timestamp.after.max.ms settings #191

Open nickdella opened 5 months ago

nickdella commented 5 months ago

When running topicctl check, we're getting drift-related failures for two settings on a particular topic:

[2024-04-17 09:47:27]  INFO Check failed for topic <topic> (cluster=<cluster>, env=staging):
-----------------------------+----+----------------------------------------------------------------------------------------------------------------------------------
             NAME            | OK |                                                             DETAILS
-----------------------------+----+----------------------------------------------------------------------------------------------------------------------------------
  config correct             | ✓  |
  configs consistent         | ✓  |
  topic exists               | ✓  |
  config settings correct    | ✗  | 2 keys have different values between cluster and topic config: [message.timestamp.after.max.ms message.timestamp.before.max.ms]
  replication factor correct | ✓  |
  partition count correct    | ✓  |
  throttles clear            | ✓  |
  replicas in-sync           | ✓  |

Our topic configuration YAML does not explicitly configure these settings. From what I can tell, these happen to be fairly new settings introduced in Kafka 3.6 (Oct 2023). I noticed a similar issue in the Confluent Terraform provider. Perhaps these settings need to be added to topicctl?

Additional details:

confluent kafka topic describe reports the following values for these settings (these are the defaults):

message.timestamp.after.max.ms 9223372036854775807
message.timestamp.before.max.ms 9223372036854775807

When running apply, I see the following output:

[2024-04-17 10:32:52]  WARN Found 2 key(s) set in cluster but missing from config:
----------------------------------+----------------------
                KEY               |    CLUSTER VALUE
----------------------------------+----------------------
  message.timestamp.before.max.ms | 9223372036854775807
  message.timestamp.after.max.ms  | 9223372036854775807
----------------------------------+----------------------
These will be left as-is.

Running check after the apply results in the same output as above.