search

getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
136 stars 39 forks source link

[HTTP-37] - Add support for Flink 1.16. #49

Closed kristoffSC closed 1 year ago

kristoffSC commented 1 year ago

Description

It was not possible to use HTTP connector with Flink 1.16. This PR allows to use HTTP connector with Flink 1.15 and 1.16. Additionally extra GitHub builds were added to make sure our connector can be used with different Flink versions.

Detailed description of the root cause and fix.

The https://issues.apache.org/jira/browse/FLINK-28807 introduced in Flink 1.16 changed the way how SerializationSchema and DeserializationSchema objects should be initialized. The change that impacted HTTP connector was change made in Flink's JsonRowDataSerializationSchema. Before FLINK-28807 ObjectMapper instance for JsonRowDataSerializationSchema was initialized as class field member, where in Flink 1.16 ObjecTMaper is initialized in open method which we did not call.

Proposed change simply ensure that open method will be called for every SerializationSchema and DeserializationSchema schema used by connector.

Public API was changed but with backward compatibility, although @Deprecated annotation was added to one of the SinkBuilder methods. Also the new public interface SchemaLifecycleAwareElementConverter was added. This interface allows to create Flink's ElementConverter on which open method will be called. This method will be called by HTTP Sink builder ensuring that Schema Serialziation and Deserialization object will be initialized properly for Flink 1.16.

Resolves https://github.com/getindata/flink-http-connector/issues/37

PR Checklist
github-actions[bot] commented 1 year ago
File Coverage [82.93%] :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
Utils.java 55% :x:
Total Project Coverage 93.64% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [82.93%] :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
Utils.java 55% :x:
Total Project Coverage 93.64% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [82.93%] :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
Utils.java 55% :x:
Total Project Coverage 93.64% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
Utils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
Utils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
Utils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
SerializationSchemaUtils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
SerializationSchemaUtils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
SerializationSchemaUtils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
SerializationSchemaUtils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
SerializationSchemaUtils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
SerializationSchemaUtils.java 55% :x:
Total Project Coverage 93.65% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [86.55%] :green_apple:
HttpSinkBuilder.java 100% :green_apple:
HttpDynamicSink.java 95.58% :green_apple:
HttpTableLookupFunction.java 83.78% :green_apple:
SerializationSchemaElementConverter.java 83.33% :green_apple:
GenericJsonQueryCreator.java 82.5% :green_apple:
HttpSinkInternal.java 72.73% :green_apple:
SerializationSchemaUtils.java 55% :x:
Total Project Coverage 93.65% :green_apple: