search

justonedb / kafka-sink-pg-json

Kafka sink connector for streaming messages to PostgreSQL
MIT License
91 stars 31 forks source link

Avro Issue #4

Open Oktaz opened 8 years ago

Oktaz commented 8 years ago

When using Avro messages on the topic I'm getting the following message:

java.lang.RuntimeException: Invalid syntax : ^ org.apache.kafka.connect.data.Struct@add32a01

Not sure if this is a bug or that the connector doesn't support Avro.

Props: key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://localhost:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://localhost:8081

internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false`

duncanpauly commented 8 years ago

Hi Michael,

You probably need the following setting in the connector property file:

value.converter=org.apache.kafka.connect.storage.StringConverter

Thanks Duncan

On Fri, Aug 19, 2016 at 9:00 PM, Michael notifications@github.com wrote:

When using Avro messages on the topic I'm getting the following message:

java.lang.RuntimeException: Invalid syntax : ^ org.apache.kafka.connect.data.Struct@add32a0 https://github.com/org.apache.kafka.connect.data.Struct/kafka-sink-pg-json/commit/add32a01

Not sure if this is a bug or that the connector doesn't support Avro.

Props: key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://localhost:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://localhost:8081

internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false`

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/justonedb/kafka-sink-pg-json/issues/4, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-W46WoFK9X5oHUifAWadgIhlkgpEqLks5qhgt5gaJpZM4JozjW .

decoursin commented 7 years ago

While my topic is using avro, I've tried a few different combinations, and none of them work. For example, when the value.converter=org.apache.kafka.connect.storage.StringConverter:

[2017-01-22 00:46:14,023] ERROR Task justone-kafka-sink-pg-json-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSinkTask:404)
java.lang.RuntimeException: Invalid syntax :  ^ myi�p=
ף
@��(\��@ @@date time
    at com.justone.json.Parser.parseRoot(Parser.java:439)
    at com.justone.json.Parser.parse(Parser.java:134)
    at com.justone.kafka.sink.pg.json.PostgreSQLSinkTask.put(PostgreSQLSinkTask.java:320)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:384)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:240)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:172)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:143)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2017-01-22 00:46:14,026] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerSinkTask:405)