Aiven-Open / http-connector-for-apache-kafka

Apache Kafka Connect sink connector for HTTP
Apache License 2.0
96 stars 42 forks source link

How can I get the connector to submit the follow curl? #170

Open alberttwong opened 1 year ago

alberttwong commented 1 year ago

There are not enough examples of how to get the http connector to work. Best example I could find is https://redpanda.com/blog/unify-log-data-parseable-redpanda

Here's what I need to submit

curl -v --location-trusted -u <username>:<password> -H "strict_mode: true" \
    -H "format: json" -H "jsonpaths: [\"$.name\", \"$.code\"]" \
    -H "columns: city,tmp_id, id = tmp_id * 100" \
    -T example2.json -XPUT \
    http://someserver:<443>/api/test_db/table2/_stream_load

I need multiple headers to be submitted but the following won't work since http.headers.additional is a String.

{
  "name": "http-sink-test",
  "config": {
    "connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
    "topics.regex": "m1",
    "http.authorization.type": "oauth2",
    "oauth2.client.id": "username",
    "oauth2.client.secret": "password",
    "http.headers.additional": [
      {
        "format": "json"
      },
      {
        "jsonpaths": "[\"$.userid\",\"$.time\",\"$._time\",\"$.remote_user\",\"$.ip\",\"$.request\",\"$.status\",\"$.bytes\",\"$.referrer\",\"$.agent\"]"
      }
    ],
    "http.url": "http://https://6dihxzedg.cloud-app.celerdata.com/api/demo/visits/_stream_load",
    "batching.enabled": true,
    "batch.max.size": 2,
    "tasks.max": "1",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter"
  }
}

Any advice?