getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
136 stars 39 forks source link

Possible to merge http request params with http response params? #65

Closed awakelee closed 9 months ago

awakelee commented 10 months ago

For example:

CREATE TABLE Customers (
    id STRING,
    id2 STRING,
    msg STRING,
    uuid STRING,
    details ROW<
      isActive BOOLEAN,
      nestedDetails ROW<
        balance STRING
      >
    >
) WITH (
'connector' = 'rest-lookup'
)
CREATE TABLE Orders (
    id STRING,
    id2 STRING,
    proc_time AS PROCTIME()
) WITH (
'connector' = 'datagen'
);

I need the require param id and id2 from table Customers instead of Orders:

SELECT c.id, c.id2, c.msg, c.uuid, c.isActive, c.balance FROM Orders AS o 
JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c ON o.id = c.id AND o.id2 = c.id2

I find that com.getindata.connectors.http.internal.table.lookup.JavaNetHttpPollingClient#processHttpResponse just parse http response data to RowData. If the response data does not contain id or id2, the RowData returned will contains null column.

kristoffSC commented 10 months ago

Hi @awakelee Very sorry about the wait. To be honest I don't think I understand what seems to be a problem here.

if the response data does not contain id or id2, the RowData returned will contains null column.

yes, what else would you want to have under c.id and c.id2 if not values returned by the external system? If the external system returned null for those values, this is what will you have.

JavaNetHttpPollingClient#processHttpResponse just parse http response data to RowData.

Exactly, this is its soil purpose.