Open alberttwong opened 1 year ago
There are not enough examples of how to get the http connector to work. Best example I could find is https://redpanda.com/blog/unify-log-data-parseable-redpanda
Here's what I need to submit
curl -v --location-trusted -u <username>:<password> -H "strict_mode: true" \ -H "format: json" -H "jsonpaths: [\"$.name\", \"$.code\"]" \ -H "columns: city,tmp_id, id = tmp_id * 100" \ -T example2.json -XPUT \ http://someserver:<443>/api/test_db/table2/_stream_load
I need multiple headers to be submitted but the following won't work since http.headers.additional is a String.
{ "name": "http-sink-test", "config": { "connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector", "topics.regex": "m1", "http.authorization.type": "oauth2", "oauth2.client.id": "username", "oauth2.client.secret": "password", "http.headers.additional": [ { "format": "json" }, { "jsonpaths": "[\"$.userid\",\"$.time\",\"$._time\",\"$.remote_user\",\"$.ip\",\"$.request\",\"$.status\",\"$.bytes\",\"$.referrer\",\"$.agent\"]" } ], "http.url": "http://https://6dihxzedg.cloud-app.celerdata.com/api/demo/visits/_stream_load", "batching.enabled": true, "batch.max.size": 2, "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter" } }
Any advice?
There are not enough examples of how to get the http connector to work. Best example I could find is https://redpanda.com/blog/unify-log-data-parseable-redpanda
Here's what I need to submit
I need multiple headers to be submitted but the following won't work since http.headers.additional is a String.
Any advice?