search

getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
150 stars 43 forks source link

ESP-98_SinkParameters - pass headers via properties to Http client #12

Closed kristoffSC closed 2 years ago

kristoffSC commented 2 years ago

Add HTTP Sink client header configuration via properties.

It is possible to set HTTP headers that will be added to HTTP request send by sink connector. Headers are defined via property key gid.connector.http.sink.header.HEADER_NAME = header value for example: gid.connector.http.sink.header.X-Content-Type-Options = nosniff. Properties can be set via Sink builder or Property object:

HttpSink.<String>builder()
      .setEndpointUrl("http://example.com/myendpoint")
      .setElementConverter(
          (s, _context) -> new HttpSinkRequestEntry("POST", s.getBytes(StandardCharsets.UTF_8)))
      .setProperty("gid.connector.http.sink.header.X-Content-Type-Options", "nosniff")
      .build();

or

Properties properties = Properties();
properties.setProperty("gid.connector.http.sink.header.X-Content-Type-Options", "nosniff");

HttpSink.<String>builder()
      .setEndpointUrl("http://example.com/myendpoint")
      .setElementConverter(
          (s, _context) -> new HttpSinkRequestEntry("POST", s.getBytes(StandardCharsets.UTF_8)))
      .setProperties(properties)
      .build();

In Table/SQL API, headers can be set using http sink table DDL. In example below, HTTP request done for http table will contain three headers:

CREATE TABLE http (
  id bigint,
  some_field string
) WITH (
  'connector' = 'http-sink',
  'url' = 'http://example.com/myendpoint',
  'format' = 'json',
  'gid.connector.http.sink.header.Origin' = '*',
  'gid.connector.http.sink.header.X-Content-Type-Options' = 'nosniff',
  'gid.connector.http.sink.header.Content-Type' = 'application/json'
)
PR Checklist
github-actions[bot] commented 2 years ago
File Coverage [93.19%] :green_apple:
HttpSink.java 100% :green_apple:
HttpSinkRequestEntry.java 100% :green_apple:
ConfigUtils.java 98.04% :green_apple:
JavaNetSinkHttpClient.java 96.63% :green_apple:
HttpDynamicSink.java 95.94% :green_apple:
HttpSinkBuilder.java 94.17% :green_apple:
HttpSinkWriterStateSerializer.java 93.55% :green_apple:
ConfigException.java 90.48% :green_apple:
HttpSinkInternal.java 69.32% :green_apple:
Total Project Coverage 92.71% :green_apple:
github-actions[bot] commented 2 years ago
File Coverage [94.24%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpSinkRequestEntry.java 100% :green_apple:
ConfigUtils.java 98.47% :green_apple:
JavaNetSinkHttpClient.java 96.63% :green_apple:
HttpSinkWriterStateSerializer.java 93.55% :green_apple:
HttpDynamicSink.java 92.52% :green_apple:
ConfigException.java 90.48% :green_apple:
HttpSinkInternal.java 69.32% :green_apple:
Total Project Coverage 92.87% :green_apple:
github-actions[bot] commented 2 years ago
File Coverage [94.24%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpSinkRequestEntry.java 100% :green_apple:
ConfigUtils.java 98.47% :green_apple:
JavaNetSinkHttpClient.java 96.63% :green_apple:
HttpSinkWriterStateSerializer.java 93.55% :green_apple:
HttpDynamicSink.java 92.52% :green_apple:
ConfigException.java 90.48% :green_apple:
HttpSinkInternal.java 69.32% :green_apple:
Total Project Coverage 92.87% :green_apple:
github-actions[bot] commented 2 years ago
File Coverage [94.23%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpSinkRequestEntry.java 100% :green_apple:
ConfigUtils.java 98.45% :green_apple:
JavaNetSinkHttpClient.java 96.63% :green_apple:
HttpSinkWriterStateSerializer.java 93.55% :green_apple:
HttpDynamicSink.java 92.52% :green_apple:
ConfigException.java 90.48% :green_apple:
HttpSinkInternal.java 69.32% :green_apple:
Total Project Coverage 92.87% :green_apple:
github-actions[bot] commented 2 years ago
File Coverage [94.15%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpSinkRequestEntry.java 100% :green_apple:
ConfigUtils.java 98.34% :green_apple:
JavaNetSinkHttpClient.java 96.63% :green_apple:
HttpSinkWriterStateSerializer.java 93.55% :green_apple:
HttpDynamicSink.java 92.52% :green_apple:
ConfigException.java 90.48% :green_apple:
HttpSinkInternal.java 69.32% :green_apple:
Total Project Coverage 92.82% :green_apple:
github-actions[bot] commented 2 years ago
File Coverage [94.61%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpSink.java 100% :green_apple:
HttpDynamicTableSinkFactory.java 100% :green_apple:
HttpSinkRequestEntry.java 100% :green_apple:
ConfigUtils.java 98.34% :green_apple:
JavaNetSinkHttpClient.java 96.63% :green_apple:
HttpDynamicSink.java 95.21% :green_apple:
HttpSinkWriterStateSerializer.java 93.55% :green_apple:
ConfigException.java 90.48% :green_apple:
HttpSinkInternal.java 69.32% :green_apple:
Total Project Coverage 92.99% :green_apple: