agavra
commented
3 years ago Hello @maeh2k the fact that INSERT INTO ... VALUES is failing is expected (as outlined in the other issue you linked #5801) but that you cannot perform a JOIN is unexpected. Can you give us the information regarding that specific error (with the JOIN)?
I attempted to reproduce this locally and could not:
$ cat test.avsc
{
"schema": "{\r\n \"namespace\": \"com.philip.test\",\r\n \"type\": \"record\",\r\n \"name\": \"TestSchema\",\r\n \"doc\": \"Test schema to reproduce bug with minimal schema.\",\r\n \"fields\": [\r\n {\r\n \"name\": \"someAttribute\",\r\n \"type\": \"string\",\r\n \"doc\": \"Some non-nullable attribute in a schema with FULL_TRANSITIVE compatibility.\"\r\n }\r\n ]\r\n}"
}
$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data "@test.avsc" http://localhost:8081/subjects/test-value/versions
{"id":1}
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility": "FULL_TRANSITIVE"}' http://localhost:8081/config/test-value
{"compatibility":"FULL_TRANSITIVE"}%Insert some data:
./bin/kafka-avro-console-producer --bootstrap-server localhost:9092 --topic test --property value.schema='{ 24s 10:35:11
"namespace": "com.philip.test",
"type": "record",
"name": "TestSchema",
"doc": "Test schema to reproduce bug with minimal schema.",
"fields": [
{
"name": "someAttribute",
"type": "string",
"doc": "Some non-nullable attribute in a schema with FULL_TRANSITIVE compatibility."
}
]
}' --property key.serializer=org.apache.kafka.common.serialization.StringSerializer --property parse.key=true --property key.separator=,
1,{"someAttribute": "foo"}Then I spun up ksql:
ksql> create table test (id varchar primary key) with (kafka_topic='test', value_format='avro', partitions=2);
Message
---------------
Table created
---------------
ksql> create stream s (id varchar key, col int) with (kafka_topic='s', value_format='avro', partitions=2);
Message
----------------
Stream created
----------------
ksql> create stream j as select * from s join test on s.id = test.id;
Message
--------------------------------
Created query with ID CSAS_J_9
--------------------------------
ksql> insert into s (id, col) values ('1', 1);I confirmed that the INSERT VALUES call does not work:
ksql> insert into test (id, someattribute) values ('1', 'bar');
Failed to insert values into 'TEST'. Could not serialize value: [ 'bar' ]. Error serializing message to topic: test. Failed to serialize Avro data from topic test :But the join does work as it is confirmed that it has data in it (from the manual insert):
ksql> SET 'auto.offset.reset'='earliest';
Successfully changed local property 'auto.offset.reset' to 'earliest'. Use the UNSET command to revert your change.
ksql> select * from j emit changes;
+-------------------+-------------------+-------------------+-------------------+
|S_ID |S_COL |TEST_ID |TEST_SOMEATTRIBUTE |
+-------------------+-------------------+-------------------+-------------------+
|1 |1 |1 |foo |
philipschm1tt
The Problem I create a table from an existing topic using Avro with FULL_TRANSITIVE compatibility – with full schema inference. Creating the table is successful, but when inserting data or just joining it to another topic, there is an error: "Schema being registered is incompatible with an earlier schema for subject"
Disclaimer This might be a duplicate of #5801 @MichaelDrogalis suggested to open an issue so that this gets another look. The error is the same. The original schema is also converted to a KSQL schema and checked for compatibility. In that issue the problem occurs when creating a stream – as opposed to using a table that was already created. But the issue mentions that "FULL_TRANSITIVE makes it almost certain that ksql registered schemas will not be compatible with previous schemas because the fields are nullable."
To Reproduce I’m using ksqlDB 0.13 in docker against Confluent Cloud incl. the Confluent Cloud Schema Registry.
Results in:
Additional context Originally, I did not even want to insert data into the table. I wanted to join the table with another table into a new topic. So I was surprised when it looked like ksqlDB was trying to register a new schema on my input topic.
My workaround was to create a STREAM from the topic instead, and then create a new table (new topic) from that stream.
Our schemas use FULL_TRANSITIVE compatibility by default. If this issue occurs with any of our topics, we do not want to work around the issue by copying the same data to a new topic every time.