search

getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
150 stars 43 forks source link

[HTTP-38] - Fixing issue with handling Http timeout where Event task was not removed from Flink's AsyncIO operator internal queue. #39

Closed kristoffSC closed 1 year ago

kristoffSC commented 1 year ago

Fixes https://github.com/getindata/flink-http-connector/issues/38

The Async version of LookupSource is guarded by two timeout timers.

First one is coming from Flink's AsyncIO operator with default value set to 3 minutes. The second one is set by connect code on HTTP request and currently is set to 2 minutes.

In case of HTTP request timeout, the the task representing this request is not removed from Flink's AsyncIO intenral task queue. This will eventually fill the entire queue (default value 100) and created backpresure on upstream elements.

In case of any timeout or exception coming from Http client, task should be removed from Flink's queue.

In this PR we make sure that for case when we had an Exception from Http Client, we will call resultFuture.complete on CompletableFuture commign from Flink Runtime in AsyncHttpTableLookupFunction.eval method.

this PR also adds a custom thread pools for Http Sink calls, where previously JVM shared ForkJoinPool was used.

At the end, various settings for HTTP and Flink AsyncIO timeouts were extracted to properties.

github-actions[bot] commented 1 year ago
File Coverage [97.15%] :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
Total Project Coverage 94.39% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.87%] :green_apple:
HttpSinkWriter.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 97.66% :green_apple:
JavaNetHttpClientFactory.java 95.43% :green_apple:
BodyBasedRequestFactory.java 86.36% :green_apple:
GetRequestFactory.java 84.62% :green_apple:
HttpSinkInternal.java 70.54% :green_apple:
Total Project Coverage 94.5% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.94%] :green_apple:
HttpSinkWriter.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 97.66% :green_apple:
JavaNetHttpClientFactory.java 95.43% :green_apple:
BodyBasedRequestFactory.java 86.36% :green_apple:
GetRequestFactory.java 84.62% :green_apple:
HttpSinkInternal.java 70.54% :green_apple:
Total Project Coverage 94.48% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.95%] :green_apple:
HttpSinkWriter.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 97.66% :green_apple:
JavaNetHttpClientFactory.java 95.43% :green_apple:
BodyBasedRequestFactory.java 86.36% :green_apple:
GetRequestFactory.java 84.62% :green_apple:
HttpSinkInternal.java 70.54% :green_apple:
Total Project Coverage 94.48% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [93.63%] :green_apple:
HttpSinkWriter.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
ConfigUtils.java 97.97% :green_apple:
JavaNetSinkHttpClient.java 97.66% :green_apple:
JavaNetHttpClientFactory.java 95.43% :green_apple:
JavaNetHttpPollingClient.java 94.37% :green_apple:
SecurityContext.java 89.29% :green_apple:
BodyBasedRequestFactory.java 86.36% :green_apple:
GetRequestFactory.java 84.62% :green_apple:
Slf4JHttpLookupPostRequestCallback.java 82.88% :green_apple:
HttpSinkInternal.java 70.54% :green_apple:
Total Project Coverage 94.24% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [93.63%] :green_apple:
HttpSinkWriter.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
ConfigUtils.java 97.97% :green_apple:
JavaNetSinkHttpClient.java 97.66% :green_apple:
JavaNetHttpClientFactory.java 95.43% :green_apple:
JavaNetHttpPollingClient.java 94.37% :green_apple:
SecurityContext.java 89.29% :green_apple:
BodyBasedRequestFactory.java 86.36% :green_apple:
GetRequestFactory.java 84.62% :green_apple:
Slf4JHttpLookupPostRequestCallback.java 82.88% :green_apple:
HttpSinkInternal.java 70.54% :green_apple:
Total Project Coverage 94.24% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [93.63%] :green_apple:
HttpSinkWriter.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
ConfigUtils.java 97.97% :green_apple:
JavaNetSinkHttpClient.java 97.66% :green_apple:
JavaNetHttpClientFactory.java 95.43% :green_apple:
JavaNetHttpPollingClient.java 94.37% :green_apple:
SecurityContext.java 89.29% :green_apple:
BodyBasedRequestFactory.java 86.36% :green_apple:
GetRequestFactory.java 84.62% :green_apple:
Slf4JHttpLookupPostRequestCallback.java 82.88% :green_apple:
HttpSinkInternal.java 70.54% :green_apple:
Total Project Coverage 94.24% :green_apple: