Closed marystory closed 5 days ago
Hello @marystory, thank you for trying out schema evolution feature!
I don't see any error within your config and v2.2.2 should definitely support schematization. Please provide information on:
Hello @sfc-gh-mbobowski
I did run into the same issue today. It happens when I create the table (with record_metadata only) with a tag before the connector is setup. When the table does not exists, it works.
I am using:
Best, Danny
Here the log
2024-07-15 13:26:39,237 INFO [connect-structured|task-2] [SF_KAFKA_CONNECTOR] Successfully called insertRows for channel:T.DATA.STRUCTURED.STRUCTURED_7, buffer:StreamingBuffer{numOfRecords=5, bufferSizeBytes=3829, firstOffset=0, lastOffset=4}, insertResponseHasErrors:true, needToResetOffset:false (com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel) [task-thread-connect-structured-2]
2024-07-15 13:26:39,238 ERROR [connect-structured|task-2] WorkerSinkTask{id=connect-structured-2} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. Error: Error inserting Records using Streaming API with msg:The given row cannot be converted to the internal format: Extra columns: ["RECORDID", "TODOS"]. Columns not present in the table shouldn't be specified, rowIndex:0 (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connect-structured-2]org.apache.kafka.connect.errors.DataException: Error inserting Records using Streaming API with msg:The given row cannot be converted to the internal format: Extra columns: ["RECORDID", "TODOS"]. Columns not present in the table shouldn't be specified, rowIndex:0
at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.handleInsertRowsFailures(TopicPartitionChannel.java:791)
at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.insertBufferedRecords(TopicPartitionChannel.java:557)
at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.insertBufferedRecordsIfFlushTimeThresholdReached(TopicPartitionChannel.java:526)
at com.snowflake.kafka.connector.internal.streaming.SnowflakeSinkServiceV2.insert(SnowflakeSinkServiceV2.java:293) at com.snowflake.kafka.connector.SnowflakeSinkTask.put(SnowflakeSinkTask.java:304)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:581)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:333)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:234)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:203)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)Caused by: net.snowflake.ingest.utils.SFException: The given row cannot be converted to the internal format: Extra columns: ["RECORDID", "TODOS"]. Columns not present in the table shouldn't be specified, rowIndex:0
at net.snowflake.ingest.streaming.internal.AbstractRowBuffer.verifyInputColumns(AbstractRowBuffer.java:424)
at net.snowflake.ingest.streaming.internal.AbstractRowBuffer$ContinueIngestionStrategy.insertRows(AbstractRowBuffer.java:162)
at net.snowflake.ingest.streaming.internal.AbstractRowBuffer.insertRows(AbstractRowBuffer.java:469)
at net.snowflake.ingest.streaming.internal.ParquetRowBuffer.insertRows(ParquetRowBuffer.java:37)
at net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal.insertRows(SnowflakeStreamingIngestChannelInternal.java:387)
at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel$InsertRowsApiResponseSupplier.get(TopicPartitionChannel.java:665)
at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel$InsertRowsApiResponseSupplier.get(TopicPartitionChannel.java:625) at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at dev.failsafe.internal.FallbackExecutor.lambda$apply$0(FallbackExecutor.java:51)
at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:182)
at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:438)
at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.insertRowsWithFallback(TopicPartitionChannel.java:619)
at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.insertBufferedRecords(TopicPartitionChannel.java:545) ... 14 more
@dthuering Since you created the table outside of the kafka connector, did you ALTER TABLE
@sfc-gh-xhuang No, I did not. It works when I do. Thank you! 👍
@marystory Can you also recheck if the table was created with ENABLE_SCHEMA_EVOLUTION?
Thanks for reviewing
@sfc-gh-xhuang Yes, as mentioned above in my post, I get the error in both of these scenarios
@sfc-gh-mbobowski the error posted above is from the connector logs. Here is the rest of the log. No data was loaded into snowflake.
I am planning to test it on confluent kafka cluster in the next few days to see if I have similar issues or not.
@marystory thank you for posting the logs.
I investigated the code deeply and what I see is that connector behaves as if the schema evolution feature is not enabled. The error with extra columns should be caught and trigger ALTER TABLE
statement but it does not happen. I will introduce additional logs in the schematization code because right now logging is very poor and we are practically blind even on a debug level.
What I would do is:
snowflake.enable.schematization
. There should be a log that confirms it is set to true.If you find anything new I will try to help. The version with additional logging will most probably be released in the next month.
@marystory could you try it out with 2.4.0 release? It contains more verbose logging.
Closing due to inactivity.
I am testing the schematization feature for the Snowpipe Streaming ingestion method locally. I have tried it with JSON and Avro value serialization. In both cases, the destination table does not exist and is supposed to be created by the Kafka connector.
I get the error below:
ERROR [snowflake-sink-connector|task-0] [SF_KAFKA_CONNECTOR] Insert Row Error message:The given row cannot be converted to the internal format: Extra columns: ["EMAIL", "AGE", "NAME"]. Columns not present in the table shouldn't be specified, rowIndex:1 (com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel:94)
Setup: kafka_2.13-3.2.3 : for kafka connect snowflake-kafka-connector-2.2.2
Schematization should be able to create the table with my schema. So I am confused about the error I got. I get the same error even when I create the destination table manually and set its ENABLE_SCHEMA_EVOLUTION to true.
Any guidance on whether I am missing a setup or using wrong versions? I am using redpanda cluster and kafka connect.
Thanks