Open satpalgrewal opened 6 months ago
Unfortunately, this does sound like a networking issue, not much I can do to help that. I'd recommend running the tool as close to the cluster here to avoid unnecessary network hops.
Hi Abraham ,
We have managed to find a temporary solution regarding the network issues that satpal posted , but currently we are testing the tool against the new parameters that you have introduced -allowList & -disallowList and whenever we ran it against the prefix for both public and private topics that we wanted to test against , but it doesn't show anything and outputs 0 to be removed .
Our naming conventions for public topics are sainsburys.data. and for private we have sainsburys.applications. / sainsburys.teams* so we want to capture a list for both separately .
The Command i have been using : ~/go/bin/kafka-idle-topics -bootstrap-servers lkc-5vq7q-41jq3.eu-west-1.aws.glb.confluent.cloud:9092 -username ????? -password ???? -kafkaSecurity plain_tls -idleMinutes 43800 -allowList 'sainsburys.teams.'
The Output i get : 2024/02/08 15:28:30 Loading Topics... 2024/02/08 15:28:35 Evaluating Topics that haven't been produced to since... 2024-01-09 05:28:35.745909 +0000 GMT m=-2627992.892874499 2024/02/08 15:28:36 Evaluating Topics without active Consumer Groups... 2024/02/08 15:31:05 Evaluating Topics without anything in them... 2024/02/08 15:31:05 Done! You can delete 0 topics and 0 partitions! A list of found idle topics is available at: /~idleTopics.txt
Any suggestions of what we can do , to be able to run the test against both public and private topics .
Hi @SandeepSehra, allowlist/disallow lists are literal lists, and do not allow prefixes. If you are hoping to filter out those topics, you have to list out their literal names.
@abraham-leal thanks for that we managed to use the -hideTopicsPrefixes parameter instead for the above scenario in which we had another question of how back can we track the topics being idle for as currently we are using this parameter -idleMinutes and set the value around 1 month , in which can we set this as 3 months for an example or further .
@SandeepSehra Glad -hideTopicsPrefixes
helped your usecase. I built it in for something different but its nice to see it evolve :) It should not be a problem to use -idleMinutes
for that length of time. The parameter uses the minutes to calculate the amount of time it must have passed since the last message stored on the topic, so it isn't any more or less efficient to use dates further back.
Hi Abraham
I am testing the tool and having issues which may be network related but need some way of confirming this
We have a private cluster which is connected to confluent via AWS private links and on-prem AWS Transit gateway and to laptop via VPN.
Command: ~/go/bin/kafka-idle-topics -bootstrap-servers ????.eu-west-1.aws.glb.confluent.cloud:9092 -username ??? -password ?? -kafkaSecurity plain_tls -idleMinutes 43800
Outputs - Attempt 1 2024/01/26 14:25:37 Loading Topics... 2024/01/26 14:25:37 Could not reach cluster within the last 30 seconds. Is the configuration correct? read tcp 172.31.193.152:59524->10.14.163.147:9092: read: connection reset by peer
Output - Attempt 2 2024/01/26 15:47:31 Loading Topics... 2024/01/26 15:47:34 Evaluating Topics that haven't been produced to since... 2023-12-27 05:47:34.83183 +0000 GMT m=-2627996.097033499 2024/01/26 15:47:35 Could not consume from topic: read tcp 172.31.193.152:49521->10.14.163.179:9092: read: connection reset by peer
From the same laptop I can confirm I can consume messages from a topic using the command kafka-console-consumer --bootstrap-server??.eu-west-1.aws.glb.confluent.cloud:9092 --consumer.config command-nonprod.properties --topic ??????--from-beginning
Any suggests ?