Some zookeeper znodes can remain after database or table are dropped

Version: 23.10.4.25 + zookeeper 3.8.2

It seems with the default replica path, if a table is created and dropped, one or more znodes associated with that table may not get deleted.

This can be problematic if, for example, automated tests are creating and deleting unique databases and tables, causing znodes to grow. It would require manual znode cleanup from time to time, which is not ideal.

Is this intended behaviour, if so could the reasoning be explained? Is there any setting that can help with clean up of these stale znodes?

To reproduce:

Create database: create database test on cluster default

Create table:

CREATE TABLE IF NOT EXISTS test.mytable ON CLUSTER default
(
   a Int32,
   d DateTime64 DEFAULT now(),
) ENGINE = ReplicatedMergeTree()
PARTITION BY toYYYYMM(d)
ORDER BY (d)

Internally this uses the default replica path: ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')

Drop the database: drop database test on cluster default sync

The /clickhouse/tables/{uuid} znode will not get deleted:

getAllChildrenNumber /clickhouse/tables
1

SELECT *
FROM system.zookeeper
WHERE path = '/clickhouse/tables'
FORMAT Vertical

Query id: 75e82258-60db-4bd6-bcae-4745023cbcaf

Row 1:
──────
name:  522b38bb-a75a-4fc3-b223-b6f0037ff78d
value:
path:  /clickhouse/tables

ClickHouse / ClickHouse

Some zookeeper znodes can remain after database or table are dropped #58964