search

haoch / flink-siddhi

A CEP library to run Siddhi within Apache Flink™ Streaming Application (Not maintained)
Apache License 2.0
243 stars 96 forks source link

Support Siddhi QueryCallback #17

Open aagupta1 opened 5 years ago

aagupta1 commented 5 years ago

Hi @haoch, Siddhi supports both StreamCallback and QueryCallback.

The flink-siddhi library only has support for StreamCallback as far as I can tell.

I'd like to be able to register a QueryCallback as well.

Do you mind if I work on this ?

haoch commented 5 years ago

What's the purpose fo QueryCallback? As Stream concept is used to bind with Flink DataStream, how should we use QueryCallback in flink?

aagupta1 commented 5 years ago

@haoch, I'm a beginner with Flink and Siddhi. Maybe I am missing something here.

My use case is this - I have multiple Siddhi queries that output results to a common Flink DataStream. I want to be able to take custom action on the query results based on the type of query.

I was thinking that if I could add a QueryCallback, I could take custom action based on the Query name.

haoch commented 5 years ago

It makes sense then. The feature sounds good. We may expect a new method: DataStream retainQuery(queryName) on SiddhiStream

aagupta1 commented 5 years ago

Thanks for the pointer. I will work on this.

haoch commented 5 years ago

Thanks @aagupta1 👍

haoch commented 5 years ago

How about the progress @aagupta1 ?

aagupta1 commented 5 years ago

@haoch apologies, working on this one now.

If we add a new method DataStream retainQuery(queryName) on SiddhiStream, are you expecting this method as a replacement for the returns, returnsAsMap, returnsAsRow methods. If that is true, then retainQuery would have to call tranform() to invoke the SiddhiStreamOperator.

The other way to do this is to get the Flink output data stream by calling the returns methods, and additionally also call retainQuery to get the DataStream for a particular query.

I'm guessing the first approach makes more sense. retainQuery should be able to be called independently of returns, in which case, it should be able to kickoff the execution of the SiddhiStreamOperator and set up a QueryCallback.

Your thoughts ?

haoch commented 5 years ago

I think retainQuery(queryName) will not be replacement for returns/returnsAsMap/returnsAsRow (streamId) as lots of users may not use queryName, so will should add another new API retainQuery(queryName)

aagupta1 commented 5 years ago

Ok, so we have two use cases -

  1. User wants to get output of one or more queries as a Flink DataStream and does not care about binding to a particular query. They will use returns / returnsAsMap / returnsAsRow(streamId). These methods will in turn invoke SiddhiStreamOperator to apply the transformations on the input stream(s).

  2. User wants to get output of a certain query as a Flink DataStream. They will use retainQuery(queryName). This method will in turn invoke SiddhiStreamOperator to apply the transformations on the input stream.

Makes sense ?