search

getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
150 stars 43 forks source link

LookupSource - Flink's task queue for async mode not clean up after HTTP client timeout #38

Closed kristoffSC closed 1 year ago

kristoffSC commented 1 year ago

The Async version of LookupSource is guarded by two timeout timers.

First one is coming from Flink's AsyncIO operator with default value set to 3 minutes. The second one is set by connect code on HTTP request and currently is set to 2 minutes.

In case of HTTP request timeout, the the task representing this request is not removed from Flink's AsyncIO intenral task queue. This will eventually fill the entire queue (default value 100) and created backpresure on upstream elements.

In case of any timeout or exception coming from Http client, task should be removed from Flink's queue.

Additionally there should be a way to set timeout parameters from table DDL.