search

getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
136 stars 39 forks source link

HTTP-42 Add support for Batch request submission in HTTP Sink. #58

Closed kristoffSC closed 1 year ago

kristoffSC commented 1 year ago

Description

This PR adds support for submitting events by HTTP Sink in batch mode, meaning that body of one HTTP PUT/POST requests will contain data for many processed events.

The data is represented as Json array, like so:

[
  {
    "id": 1,
    "first_name": "Ninette",
    "last_name": "Clee",
    "gender": "Female",
    "stock": "CDZI",
    "currency": "RUB",
    "tx_date": "2021-08-24 15:22:59"
  },
  {
    "id": 2,
    "first_name": "Rob",
    "last_name": "Zombie",
    "gender": "Male",
    "stock": "DGICA",
    "currency": "GBP",
    "tx_date": "2021-10-25 20:53:54"
  },
  {
    "id": 3,
    "first_name": "Adam",
    "last_name": "Jones",
    "gender": "Male",
    "stock": "DGICA",
    "currency": "PLN",
    "tx_date": "2021-10-26 20:53:54"
  }
]

This is a breaking change. Users will have to adapt to it or restore "single mode" by setting: gid.connector.http.sink.writer.request.mode = single

Resolves https://github.com/getindata/flink-http-connector/issues/42

PR Checklist
github-actions[bot] commented 1 year ago
File Coverage [97.87%] :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestSubmitter.java 95% :green_apple:
Total Project Coverage 93.68% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [89.64%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
BatchRequestSubmitter.java 94.29% :green_apple:
PerRequestSubmitter.java 93.33% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
BatchRequestSubmitterFactory.java 50% :x:
Total Project Coverage 93.19% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [93.44%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
BatchRequestSubmitter.java 93.78% :green_apple:
PerRequestSubmitter.java 93.41% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Total Project Coverage 93.83% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [93.44%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
BatchRequestSubmitter.java 93.78% :green_apple:
PerRequestSubmitter.java 93.41% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Total Project Coverage 93.83% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.58%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 94.02% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.85% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.58%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 94.02% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.85% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.58%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 94.02% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.85% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.58%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 94.02% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.85% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.58%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 94.02% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.85% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.58%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 94.02% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.85% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.5%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 93.46% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.83% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.5%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 93.46% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.83% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [94.5%] :green_apple:
SinkRequestSubmitMode.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkBuilder.java 100% :green_apple:
JavaNetHttpResponseWrapper.java 100% :green_apple:
AbstractRequestSubmitter.java 100% :green_apple:
BatchRequestSubmitterFactory.java 100% :green_apple:
JavaNetSinkHttpClient.java 100% :green_apple:
PerRequestRequestSubmitterFactory.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
JavaNetHttpClientFactory.java 95.35% :green_apple:
PerRequestSubmitter.java 94.12% :green_apple:
BatchRequestSubmitter.java 93.46% :green_apple:
HttpSinkInternal.java 75.52% :green_apple:
Slf4jHttpPostRequestCallbackFactory.java 52.94% :x:
Total Project Coverage 93.83% :green_apple: