Big Query Connect Exception ..... not able to write data to BigQuery

[mukund26]

I am having this error when i send the update avro schema to bigQuery connector:

[2018-07-02 13:23:45,421] ERROR Task failed with com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException error: Failed to write rows after BQ schema update within 5 attempts for: {datasetId=intern_practice, tableId=full_public_cons} (com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor:66)
Exception in thread "pool-5-thread-3" com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: Failed to write rows after BQ schema update within 5 attempts for: {datasetId=intern_practice, tableId=full_public_cons}
    at com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.performWriteRequest(AdaptiveBigQueryWriter.java:120)
    at com.wepay.kafka.connect.bigquery.write.row.BigQueryWriter.writeRows(BigQueryWriter.java:117)
    at com.wepay.kafka.connect.bigquery.write.batch.TableWriter.run(TableWriter.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[bob9]

I am having the same issue.. did you manage to make any headway?

The BigQuery table gets created however no data is inserted into the tables

[mukund26]

@bob9 sadly i couldn't find any way to make it work except hard coding the table format for which i was working

[Sano123456]

in my case if you use transforms it give me the same issue

transforms=InsertField
transforms.InsertField.type=org.apache.kafka.connect.transforms.InsertField$Value
transforms.InsertField.timestamp.field=fieldtime

[criccomini]

In the case of the transform, I suspect there might be issues because we're fetching schemas from the schema registry if you're transforming the message such that it no longer has the same schema, which it looks like your transform would do, then I think you'll have a schema mismatch when inserting into BQ.

I don't think there's an easy way to fix this right now, other than to manually modify the table schema in BQ once so that it matches the schema that reflects how you're transforming the data.

[Sano123456]

hi @criccomini thanks very much for your explanation. If possibile i would like to write a column timestamp with the value of ROWTIME of the kafka message

what is the best way to do it using your connector? thanks very much

[criccomini]

If it's just for a small number of tables, I suggest just manually setting up the schema in BQ yourself. The way KCBQ works is that it lazily inserts messages into BQ, and only tries to do things with the schema if the inserts fail. If you manually set up the BQ tables to have the proper schema according to the rows you're trying to insert, there should be no problem.

[Singh-Lovepreet]

Hi,i want to kcbq but is have one doubt that kcbq is just for insert data opration into big-query or also perform update operation in big query @criccomini