Topic's config can no longer be updated after being recreated

Masqueey commented 2 weeks ago

Issue submitter TODO list

[X] I've looked up my issue in FAQ
[X] I've searched for an already existing issues here
[X] I've tried running main-labeled docker image and the issue still persists there
[X] I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

After recreating a topic, its configuration settings get frozen. The error that is shown when trying to update the settings is not relevant in this case and other topics with the same config can still be updated.
Related FE issue: when settings are not frozen, they are grayed out and not clickable, but work when manually typed. When they can no longer be updated, they are black and clickable.

Expected behavior

After recreating a topic, its settings remain as configurable as before recreating it.
The FE part should be inverted, gray and not clickable when frozen, black and clickable when configurable.

Your installation details

b4ffd96
Custom company internal chart

Doubt it's relevant but some config:

kafka-ui:
# kubernetes resources
image:
repository: kafbat/kafka-ui
tag: "b4ffd96b95536ebe5891c5be99b9ce6ba7cab241"

yamlApplicationConfig:
rbac:
  roles:
    - name: "admins"
    - name: "readonly"
      permissions:
        - resource: topic
          value: ".*"
          actions: [view, messages_read]
        - resource: consumer
          value: ".*"
          actions: view

envs:
config:
  # KAFKA CLUSTER
  KAFKA_CLUSTERS_0_NAME: kafka-cluster
  KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SSL
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION: /keystore/keystore.jks
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD: REPLACE ME!
  KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS:

  # TRUSTSTORE
  KAFKA_CLUSTERS_0_SSL_TRUSTSTORE_LOCATION: /truststore/truststore.jks
  KAFKA_CLUSTERS_0_SSL_TRUSTSTORE_PASSWORD: REPLACE ME!

Steps to reproduce

Create a topic with the following config:

"config": {
"compression.type": "producer",
"leader.replication.throttled.replicas": "",
"remote.storage.enable": "false",
"message.downconversion.enable": "true",
"min.insync.replicas": "2",
"segment.jitter.ms": "0",
"local.retention.ms": "-2",
"cleanup.policy": "compact",
"flush.ms": "9223372036854775807",
"follower.replication.throttled.replicas": "",
"segment.bytes": "1073741824",
"retention.ms": "864000000",
"flush.messages": "9223372036854775807",
"remote.log.msk.disable.policy": "None",
"message.format.version": "3.0-IV1",
"max.compaction.lag.ms": "9223372036854775807",
"file.delete.delay.ms": "60000",
"max.message.bytes": "1048588",
"min.compaction.lag.ms": "0",
"message.timestamp.type": "CreateTime",
"local.retention.bytes": "-2",
"preallocate": "false",
"min.cleanable.dirty.ratio": "0.5",
"index.interval.bytes": "4096",
"unclean.leader.election.enable": "true",
"retention.bytes": "300000000000",
"delete.retention.ms": "86400000",
"message.timestamp.after.max.ms": "9223372036854775807",
"message.timestamp.before.max.ms": "9223372036854775807",
"segment.ms": "86400000",
"message.timestamp.difference.max.ms": "9223372036854775807",
"segment.index.bytes": "10485760"
},

Go to the topic.
(FE issue: go to "edit settings" > "add custom parameter" > delete.retention.ms > Notice it is grayed out and not clickable.)
Three dots in the top right > Recreate Topic > Yes.
Three dots in the top right > "edit settings" > "add custom parameter" > delete.retention.ms > add a '0' to the end.
Update topic
Get the 500 error

Screenshots

No response

Logs

[90091539-322]  500 Server Error for HTTP PATCH "/api/clusters/kafka-cluster/topics/<topic-name>"
│ org.apache.kafka.common.errors.InvalidConfigurationException: Invalid config value for resource ConfigResource(type=TOPIC, name='<topic-name>'): Deletion of configuration: `remote.storage.enable` is not supported for the topic <topic-name>. To disable remote storage, please set the value to `false` along with `remote.log.msk.disable.policy` configuration as `Delete`.` 
│     Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:                                                                                                                                                               │
│ Error has been observed at the following site(s):                                                                                                                                                                                        │
│     *__checkpoint ⇢ Handler io.kafbat.ui.controller.TopicsController#updateTopic(String, String, Mono, ServerWebExchange) [DispatcherHandler]                                                                                            │
│     *__checkpoint ⇢ io.kafbat.ui.config.CorsGlobalConfiguration$$Lambda$1239/0x00007ff8cc71d9e8 [DefaultWebFilterChain]                                                                                                                  │
│     *__checkpoint ⇢ io.kafbat.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]                                                                                                                                                       │
│     *__checkpoint ⇢ io.kafbat.ui.config.CustomWebFilter [DefaultWebFilterChain]                                                                                                                                                          │
│     *__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]                                                                                                                                                                       │
│     *__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]                                                                                                                                                                │
│     *__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]                                                                                                                                                                              │
│     *__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]                                                                                                                                                                  │
│     *__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]                                                                                                                                                    │
│     *__checkpoint ⇢ LogoutPageGeneratingWebFilter [DefaultWebFilterChain]                                                                                                                                                                │
│     *__checkpoint ⇢ LoginPageGeneratingWebFilter [DefaultWebFilterChain]                                                                                                                                                                 │
│     *__checkpoint ⇢ OAuth2LoginAuthenticationWebFilter [DefaultWebFilterChain]                                                                                                                                                           │
│     *__checkpoint ⇢ OAuth2AuthorizationRequestRedirectWebFilter [DefaultWebFilterChain]                                                                                                                                                  │
│     *__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]                                                                                                                                                                      │
│     *__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]                                                                                                                                                                    │
│     *__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]                                                                                                                                                     │
│     *__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]                                                                                                                                  │
│     *__checkpoint ⇢ HTTP PATCH "/api/clusters/kafka-cluster/topics/<topic-name>" [ExceptionHandlingWebHandler]

Additional context

Restarting the pod did not fix the issue, neither did retrying from another admin user account.
Only deleting and recreating the topic manually (deleting, then creating again) (with the exact same settings) resolved the issue. (As in, we could update the settings again.)
Might be relevant: the error says remote.log.msk.disable.policy configuration as Delete whereas the setting on the topic is set to None
- This is not a blocking setting normally, only when the topic gets recreated (at least, according to the error).

github-actions[bot] commented 2 weeks ago

Hi Masqueey! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues. Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.

germanosin commented 2 weeks ago

Thank you, @Masqueey, for your contribution! It appears that the default configuration property for log.msk.disable.policy via the API is set to "Delete," whereas for the manual method (could you clarify what that entails?), it’s set to "None." Would you mind sharing more details on this?

Masqueey commented 2 weeks ago

Thanks for your fast reply!

With "the manual method" I mean using our own api to copy the config, delete the topic, then create a topic with the same name and same (copied) config. Then the topic is usable again. (I assumed the "recreate topic" button in the UI did something similar.)

The default configuration property for log.msk.disable.policy via the API is set to "Delete," whereas for the manual method, it’s set to "None."

I noticed this too, however it is not an issue before I use the "recreate topic" button. The setting also remains "None" after the topic has been recreated. Updating it to something else after the recreation is not possible.

We could, of course, update our own API to set log.msk.disable.policy to 'Delete' when it is included, but even then, why would this only be an issue after recreating it? On top of that, most of our topics have this setting currently on None, so this would only be relevant for future topics.

Masqueey commented 1 week ago

To maybe aid in debugging this further, I now created a topic through the UI, noticed the remote.log.msk.disable.policy setting was already set to None, even when not explicitly added in Custom Parameters in the creation process. (Could be some AWS or MSK setting? Not sure how to check that.)

But also here, if you recreate it with the recreate button, it freezes the settings due to the above setting being None. (According to the error.)

What I tried to test here was whether this was caused by first creating the topic through a different API and then recreating it through the KafkaUI. But turns out that even when (re)creating it through the UI, the setting is set to None automatically if not set explicitly to something else.

woile commented 1 week ago

I'm having a similar issue. I cannot edit delete.retention.ms on a compacted log topic. Which is strange, because it's a setting specifically for compacted logs.

Edit: it works if I type the whole name, but selecting is disabled.

kafbat / kafka-ui