search

getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
150 stars 43 forks source link

[HTTP-34] Add support for lookup on Row type #35

Closed kristoffSC closed 1 year ago

kristoffSC commented 1 year ago

Description

This PR Resolves HTTP-34.

It adds support for SQL looup joins scenarios where ROW type column is used in JOIN condition. In that case, Lookup connector converts used columns (including nested types like Arrays, Maps Lists and ROWs) into proper http query:

  1. for GET requests it converts all fields (including ROW columns) to key=value pairs and use them in HTTP query parameters
  2. For Body based requests like POST/PUT it converts entire lookup row into JSON object and use it as query's body.

By default Flink's Json-Format is used for RowData -> Json conversion.

However user can created and use its own Serialization formats. To create a custom format user has to implement Flink's SerializationSchema and SerializationFormatFactory interfaces and register custom format factory along other factories in resources/META-INF.services/org.apache.flink.table.factories.Factory file. This is common Flink mechanism for providing custom implementations for various factories.

In order to use custom format, user has to specify option 'lookup-request.format' = 'customFormatName', where customFormatName is the identifier of our custom format factory.

Additionally, it is possible to pass query format options from table's DDL. This can be done by using option like so: 'lookup-request.format.customFormatName.cotom_formatProperty' = 'propertyValue', for example 'lookup-request.format.customFormatName.fail-on-missing-field' = 'true'. It is important that customFormatName part must match SerializationFormatFactory identifier used for custom format implementation. In this case, the fail-on-missing-field will be passed to SerializationFormatFactory::createEncodingFormat( DynamicTableFactory.Context context, ReadableConfig formatOptions) method in ReadableConfig object.

PR Checklist
github-actions[bot] commented 1 year ago
File Coverage [92.13%] :green_apple:
LookupSchemaEntryBase.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
SingleValueLookupSchemaEntry.java 94.29% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.55% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [91.53%] :green_apple:
LookupSchemaEntryBase.java 100% :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
SingleValueLookupSchemaEntry.java 77.08% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.37% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [91.53%] :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
RowDataLookupSchemaEntryBase.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
RowDataSingleValueLookupSchemaEntry.java 77.08% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.37% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [91.53%] :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
RowDataLookupSchemaEntryBase.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
RowDataSingleValueLookupSchemaEntry.java 77.08% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.37% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [91.53%] :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
RowDataLookupSchemaEntryBase.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
RowDataSingleValueLookupSchemaEntry.java 77.08% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.37% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [91.53%] :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
RowDataLookupSchemaEntryBase.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
RowDataSingleValueLookupSchemaEntry.java 77.08% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.37% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [91.53%] :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
RowDataLookupSchemaEntryBase.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
RowDataSingleValueLookupSchemaEntry.java 77.08% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.37% :green_apple:
github-actions[bot] commented 1 year ago
File Coverage [91.53%] :green_apple:
RequestFactoryBase.java 100% :green_apple:
HttpLookupTableSourceFactory.java 100% :green_apple:
AsyncHttpTableLookupFunction.java 100% :green_apple:
RowTypeLookupSchemaEntry.java 100% :green_apple:
LookupRow.java 100% :green_apple:
RowDataLookupSchemaEntryBase.java 100% :green_apple:
GenericJsonQueryCreator.java 100% :green_apple:
GenericGetQueryCreator.java 100% :green_apple:
HttpLookupTableSource.java 94.83% :green_apple:
ElasticSearchLiteQueryCreator.java 94.12% :green_apple:
JavaNetHttpPollingClient.java 93.85% :green_apple:
QueryFormatAwareConfiguration.java 92% :green_apple:
GenericJsonQueryCreatorFactory.java 91.3% :green_apple:
BodyBasedRequestFactory.java 85.71% :green_apple:
GetRequestFactory.java 83.78% :green_apple:
TableSourceHelper.java 82.93% :green_apple:
HttpTableLookupFunction.java 82.61% :green_apple:
RowDataSingleValueLookupSchemaEntry.java 77.08% :green_apple:
GenericGetQueryCreatorFactory.java 71.43% :green_apple:
PrefixedConfigOption.java 68% :green_apple:
ElasticSearchLiteQueryCreatorFactory.java 35.71% :x:
Total Project Coverage 94.37% :green_apple: