Open cxlRay opened 4 years ago
Hi @cxlRay,
You can try to minimise intermediatePersistPeriod
which is PT10M
by default and then the persisted data could apply the "vectorize": "true"
option. Hope this helps to some extent.
some issue here.
@yuanlihan thanks for your answer, minimise intermediatePersistPeriod will create much more small file
@yuanlihan thanks for your answer, minimise intermediatePersistPeriod will create much more small file
@cxlRay that's true. But the temporary persist files will be cleaned up when the hourly tasks finished. And the persisted incremental files/indexes(with extra indexes to speed up query processing) will be more efficient than the in-memory incrementalIndex
. As far as I know, when a query scan the latest in-memory incrementalIndex
(like latest 10 minutes's data), Druid processes the in-memory fact table held by incrementalIndex
row by row.
Actually, I had tried to minimise the value of maxRowsInMemory
to reduce in-memory rows but this will also introduce some overheads caused by frequently persisting.
timeseries is very slow with filter. (10-20s)
@exherb say more about your question, if your cluster run on ssd, there will be better
@exherb say more about your question, if your cluster run on ssd, there will be better
just like yuanlihan say,minimise intermediatePersistPeriod or maxRowsInMemory , add query parmeter vectorize
just like yuanlihan say,minimise intermediatePersistPeriod or maxRowsInMemory , add query parmeter vectorize
intermediatePersistPeriod: P10M maxRowsInMemory: 1000000
context: { vectorize: true }
still slow.
rows in no-rollup incremental index are stored as
ConcurrentSkipListMap<Long, ConcurrentLinkedDeque<IncrementalIndexRow>>
which seemed not easy to be fast, imho.
Anyway, can you try with coarser granularity, like 15 minutes or something?
rows in no-rollup incremental index are stored as
ConcurrentSkipListMap<Long, ConcurrentLinkedDeque<IncrementalIndexRow>>
which seemed not easy to be fast, imho.
Anyway, can you try with coarser granularity, like 15 minutes or something?
we enabled rollup to 1minutes, segment granularity by hour. Are you suggestion change segment granularity to 15 minutes?
@exherb how do you know the slow part is peon query,the query process: client -> broker -> indexing service(Peon), use monitor metric or analysis code?
@exherb how do you know the slow part is peon query,the query process: client -> broker -> indexing service(Peon), use monitor metric or analysis code?
by druid metrics: query/segment/time
/ query/time
(datasource~=.+middlemanager.+)
@exherb I mean granularity
of timeseries query.
same issue.
when send batch query to Realtime processes tasks, the performance is too bad. TPS only have 80, response time is more than one second, max response time can be 30 second and 99% response time can be 15 second. what I confused is the data of Realtime processes are in mem, why response time of query is so long?
Affected Version
v druid-0.16.1-incubating.
Description
Cluster size coordinator and overlord: 2 historical: 7 middleManager: 7 broker: 5
The testing tool is jmeter
testing result thread num: 500 TPS: 107.80 average response time: 4372ms 99% response time: 15559ms max response time: 30150ms min response time: 373
the Flame chart of Realtime task code when test
the configuration of middleManager
the configuration of Realtime tasks
about segments
my query