getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
136 stars 39 forks source link

Allow GET POST PUT parameter resolution through configuration #99

Open davidradl opened 1 month ago

davidradl commented 1 month ago

Currently the code has logic if GET then use query params otherwise use body content to resolve the parameters.

We have recently included 2 prs that allow GETs with body parameters https://github.com/getindata/flink-http-connector/pull/87 and path parameters https://github.com/getindata/flink-http-connector/pull/79

The fixes involved introducing classes that could be overridden to provide this functionality.

It would make more sense to me to allow configuration to target how parameters are resolved on a per lookup join basis so the main cases could be identified using config not requiring code to be added.

Without the new connector configuration is would default to the existing behaviour. There is an option to say useV2Parameterization, with would honour a new body parameter option than could supply the body parameter, we supply a string of json in a separate connector config (parameterised) option rather than mapping query params to a json body.

So 2 new configuration parameters:

I am hoping we can cover all basic operations with json through configuration. Customization would still be allowed for other cases.

davidradl commented 1 month ago

@kristoffSC WDYT?