gianm
commented
3 months ago Hi @ColeAtCharter. The /druid/v2/candidates API does not currently take into account segment pruning, although /druid/v2/ and /druid/v2/sql/ (the actual query APIs) do. I tried to repro the issue on /druid/v2/ and was not able to. A single segment was queried, as expected.
Could you provide some more details that could be used to repro this issue?
Here's what I tried:
I loaded data with the following SQL and rowsPerSegment: 1500 (to ensure we get multiple segments):
REPLACE INTO "wikipedia" OVERWRITE ALL
WITH "ext" AS (
SELECT *
FROM TABLE(
EXTERN(
'{"type":"http","uris":["https://druid.apache.org/data/wikipedia.json.gz"]}',
'{"type":"json"}'
)
) EXTEND ("isRobot" VARCHAR, "channel" VARCHAR, "timestamp" VARCHAR, "flags" VARCHAR, "isUnpatrolled" VARCHAR, "page" VARCHAR, "diffUrl" VARCHAR, "added" BIGINT, "comment" VARCHAR, "commentLength" BIGINT, "isNew" VARCHAR, "isMinor" VARCHAR, "delta" BIGINT, "isAnonymous" VARCHAR, "user" VARCHAR, "deltaBucket" BIGINT, "deleted" BIGINT, "namespace" VARCHAR, "cityName" VARCHAR, "countryName" VARCHAR, "regionIsoCode" VARCHAR, "metroCode" BIGINT, "countryIsoCode" VARCHAR, "regionName" VARCHAR)
)
SELECT
TIME_PARSE("timestamp") AS "__time",
"isRobot",
"channel",
"flags",
"isUnpatrolled",
"page",
"diffUrl",
"added",
"comment",
"commentLength",
"isNew",
"isMinor",
"delta",
"isAnonymous",
"user",
"deltaBucket",
"deleted",
"namespace",
"cityName",
"countryName",
"regionIsoCode",
"metroCode",
"countryIsoCode",
"regionName"
FROM "ext"
PARTITIONED BY DAY
CLUSTERED BY "page", "user"Then ran this query:
{
"queryType": "timeseries",
"dataSource": {
"type": "table",
"name": "wikipedia"
},
"intervals": {
"type": "intervals",
"intervals": [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
]
},
"filter": {
"type": "selector",
"dimension": "page",
"value": "Bailando 2015"
},
"granularity": {
"type": "all"
},
"aggregations": [
{
"type": "count",
"name": "a0"
}
],
"context": {
"bySegment": true
}
}A single segment came back in the reply:
[
{
"timestamp": "2016-06-27T00:00:00.000Z",
"result": {
"results": [
{
"timestamp": "2016-06-27T00:00:34.959Z",
"result": {
"a0": 9
}
}
],
"segment": "wikipedia_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2024-06-04T17:02:47.206Z_1",
"interval": "2016-06-27T00:00:00.000Z/2016-06-28T00:00:00.000Z"
}
}
]I also confirmed using the metric query/segment/time that only a single segment was queried.
ColeAtCharter
This is to report unexpected behavior where the broker selects too many segments to query. The identified scenario is when secondary partition information exists (eg, for range partitioning) but is not being used in conjunction with a query filter which corresponds to the secondary partitions. The result of not completely pruning segments at the broker is a detrimental impact on system operations such as for performance and cost.
Affected Version
28
Description
The /druid/v2 and /druid/v2/candidates broker endpoints are returning segments which should be filtered out based on secondary partition metadata. Returning unneeded segments for planning and executing queries will cause unnecessary I/O throughout the system, causing avoidable detriment to cost and performance.
Steps to reproduce:
Expected/observed behavior summary
Testing notes
Representation of the test setup
segments for "src1"
server segments for datasource "src1"
segment metadata - range partition column values
test query
Expected result example - /v2/candidates
If partition pruning is reflected at the broker, it would be unexpected to receive back any of "the other two segments" (regardless of server/replication)
Unexpected result example 1: /druid/v2 endpoint - extra (but not all) segments are returned
src1_2025-02-16T01:00:00.000Z_2025-02-16T02:00:00.000Z_2024-04-01T23:59:59.999Z_1Unexpected result example 2: /druid/v2/candidates endpoint - all segments are returned
src1_2025-02-16T01:00:00.000Z/2025-02-16T02:00:00.000Z_2024-04-01T23:59:59.999Z_1Summary
The broker should only select and/or query the smallest number of segments matching on datasource/time chunk and partition dimensions when it has enough has enough partition information about the used/loaded segments to prune out segments that don't match on partition dimensions (in addition to those segments not matching on datasource or time chunk)
Related documentation