[Feature] Topic level policy support in pulsarctl

[codelipenghui]

The topic level policy is planned to release 2.7.0 and there are some PRs already merged into the Pulsar master branch. The process of the topic level policy support in Pulsar can be found at https://github.com/apache/pulsar/issues/2688. It's better to support topic level policy in pulsarctl.

• [x] 1. MessageTTL [Get, Set, Remove] #348
• [x] 2. Retention Policy [Get, Set, Remove] #420
• [x] 3. Message Dispatch Rate [Get, Set, Remove] #397
• [x] 4. Backlog Quota [Get, Set, Remove] #429
• [x] 5. Persistence Policy [Get, Set, Remove] #363
• [x] 6. Max Producer [Get, Set, Remove] #353
• [x] 7. Max Consumer [Get, Set, Remove] #355
• [x] 8. Compaction Threshold [Get, Set, Remove] #430
• [x] 9. Deduplication [Get, Set, Remove] #408
• [ ] 10. Offload Policy [Get, Set, Remove]
• [x] 11. MaxUnackMessagesPerConsumer [Get, Set, Remove] #358
• [x] 12. MaxUnackMessagesPerSubscription [Get, Set, Remove] #361
• [x] 13. DelayedDeliveryPolicies [Get, Set, Remove] #374
• [x] 14. Inactive topic policy [Get, Set, Remove] #444

Append comand

• [ ] 15. clear-backlog
• [ ] 16. compaction-status
• [ ] 17. create-missed-partitions
• [x] 18. create-partitioned-topic
• [x] 19. delete-partitioned-topic
• [ ] 20. examine-messages
• [ ] 21. expire-messages-all-subscriptions
• [ ] 22. get-deduplication-snapshot-interval
• [ ] 23. get-max-consumers-per-subscription
• [ ] 24. get-max-message-size
• [ ] 25. get-max-subscriptions
• [ ] 26. get-message-by-id
• [ ] 27. get-message-id
• [ ] 28. get-partitioned-topic-metadata
• [ ] 29. get-replicator-dispatch-rate
• [ ] 30. get-subscribe-rate
• [ ] 31. get-subscription-dispatch-rate
• [ ] 32. get-subscription-types-enabled
• [x] 33. grant-permission
• [x] 34. info-internal
• [x] 35. list-partitioned-topics
• [ ] 36. partitioned-lookup
• [x] 37. partitioned-stats
• [x] 38. partitioned-stats-internal
• [x] 39. permissions
• [ ] 40. remove-deduplication-snapshot-interval
• [ ] 41. remove-max-consumers-per-subscription
• [ ] 42. remove-max-message-size
• [ ] 43. remove-max-subscriptions
• [ ] 44. remove-replicator-dispatch-rate
• [ ] 45. remove-subscribe-rate
• [ ] 46. remove-subscription-dispatch-rate
• [x] 47. revoke-permission
• [ ] 48. set-deduplication-snapshot-interval
• [ ] 49. set-max-consumers-per-subscription
• [ ] 50. set-max-message-size
• [ ] 51. set-max-subscriptions
• [ ] 52. set-replicated-subscription-status
• [ ] 53. set-replicator-dispatch-rate
• [ ] 54. set-subscribe-rate
• [ ] 55. set-subscription-dispatch-rate
• [ ] 56. set-subscription-types-enabled
• [x] 57. stats-internal
• [ ] 58. truncate
• [x] 59. update-partitioned-topic

Waiting for confirmation:

• [ ] confirmed with @codelipenghui
• [ ] confirmed with @zymap

[limingnihao]

I will add MessageTTL [Get, Set, Remove].

[limingnihao]

I will add Max Producer [Get, Set, Remove], and Max Consumer [Get, Set, Remove].

[limingnihao]

I will add MaxUnackMessagesPerConsumer [Get, Set, Remove].

[limingnihao]

I will add MaxUnackMessagesPerSubscription [Get, Set, Remove]

[limingnihao]

I will add Persistence Policy [Get, Set, Remove]

[limingnihao]

I will add Delayed Delivery Policies [Get, Set, Remove]

[limingnihao]

I will add Backlog Quota [Get, Set, Remove]

[limingnihao]

I will add Message Dispatch Rate [Get, Set, Remove]

[limingnihao]

I will add Deduplication [Get, Set, Remove]

[nodece]

I will add Retention Policy [Get, Set, Remove]

[nodece]

I will add Backlog Quota [Get, Set, Remove]

[flowchartsman]

I added Publish Rate [Get, Set, Remove], even though it is not on the list :)

[nodece]

I will add Compaction Threshold [Get, Set, Remove].

[flowchartsman]

Is there some context on the deprecation of items 10 and 14? I had been working on a PR for topic-level offload policy, and I don't see any notes on deprecation in apache/pulsar#2688

[nodece]

@flowchartsman I updated description on this issue, you can found the deprecated commands on https://github.com/apache/pulsar/blob/master/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java#L215

[flowchartsman]

I am confused, now 10 is marked as done and 14 is deprecated?

[flowchartsman]

Inactive topic policy is listed as deprecated in CmdTopics.java, but I do not see a reference to offload policy. Unless someone has completed this work already, the link on 10 above is only to the manual offload command not to offload policy. I didn't see anyone claim it, so I had started the work to support namespace/topic offload policy in my spare time, and I don't see any commits that reference it. Is this work complete? Should I abandon my partial implementation?

[nodece]

Hi @flowchartsman, 10 and 14 should be deprecated, I added deprecated commands link. apology :)

I think we don't should to implement a deprecated command, what's your opinion?

[flowchartsman]

I am not sure this is correct. On further inspection, it would appear that neither offload policy nor inactive topic policy are deprecated. The deprecated commands appear to be specified by comments above newline-separated blocks of commands. Note that there are three such comments, corresponding to three such blocks of commands. Each command in the block is then repeated explicitly in initDeprecatedCommands(), and appear to be limited to:

• enable-deduplication (notation, deprecation)
• *-max-unacked-messages-on (_but not messages-per_) (notation, deprecation)
• maxproducers (notation, deprecation)

[nodece]

You're right. We should to implement the all commands in https://github.com/apache/pulsar/blob/master/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java, and skip the deprecated commands.

If you are implementing the feature on issue, please let us known for avoid work conflicts.

[nodece]

Hello @codelipenghui and @zymap, I compared pulsar-admin topics and pulsarctl and sorted out the commands that need to be added to pulsarctl, if you are free, please help me to confirm, thanks.

[nodece]

I will implement the Inactive topic policy [Get, Set, Remove] feature.