tomasfabian / ksqlDB.RestApi.Client-DotNet

ksqlDb.RestApi.Client is a C# LINQ-enabled client API for issuing and consuming ksqlDB push and pull queries and executing statements.
MIT License
93 stars 24 forks source link

get aggregate data directly from ksqldb #15

Closed shubhamkule closed 2 years ago

shubhamkule commented 2 years ago

Hello Tomas,

we are getting data from ksqldb and trying to apply too much filter to get expected data like below, is there any way to get direct data from ksqldb like below. thanks please suggest,

ordertable


rowkey id priceArray eventtime (in unix time stamp)  
132    1  [1,5,6,3,1] 1265
134    1  [4,3,4,8,9] 1266
135    1  [1,2,6,3,6] 1267
136    1  [1,5,5,2,4] 1268
132    2  [11,5,1,3,1] 1265
134    2  [10,3,4,8,9] 1266
135    2  [2,12,6,3,6] 1267
136    2  [13,5,5,2,4] 1268

expected data

this data is group by id column and we trying to get array index wise min,max,avg  
id min         max            avg            minon           maxon
1  [1,2,4,2,1] [4,5,6,8,9]   [1.6,3.5,5.2,4,5] [1265,1267,1266,1268,1265] [1266,1265,1265,1266,1266]
2  [2,3,1,2,1] [13,12,6,8,9] [1.6,3.5,5.2,4,5] [1267,1266,1265,1268,1265] [1268,1267,1267,1266,1266]

table script


CREATE TABLE kafka_table (rowkey varchar PRIMARY KEY, id INT, eventtime BIGINT, priceArray ARRAY<double>) WITH (KAFKA_TOPIC='kafka_topic_order', FORMAT='PROTOBUF', PARTITIONS=1, REPLICAS=1);
CREATE TABLE QUERYABLE_kafka_table AS SELECT * FROM kafka_table;
tomasfabian commented 2 years ago

Hi @shubhamkule, not sure how to get the requested data with ksql (its probably even not possible). Try to use the Transform Invocation function with LATEST_BY_OFFSET or try to repost your question in Stackoverflow.

Regards Tomas