mgvinuesa
commented
1 year ago To add more information, without auto.register.schemas does not work the deserializer, ItemNotFound. So I can image that it is a problem regarding the internal azure APIs. If I disable the auto.register.schema but maintain the schema in the portal, I can see this GET request to retrieve it.
11:07:06.577 [reactor-http-nio-3] DEBUG reactor.netty.transport.TransportConnector - [a605dfa6] Connecting to [vialivre-dev-image.servicebus.windows.net/51.144.238.23:443].
11:07:06.611 [reactor-http-nio-3] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [a605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] Registering pool release on close event for channel
11:07:06.611 [reactor-http-nio-3] DEBUG reactor.netty.resources.PooledConnectionProvider - [a605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] Channel connected, now: 1 active connections, 0 inactive connections and 0 pending acquire requests.
11:07:06.715 [reactor-http-nio-3] DEBUG io.netty.handler.ssl.SslHandler - [id: 0xa605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
11:07:06.715 [reactor-http-nio-3] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [a605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] onStateChange(PooledConnection{channel=[id: 0xa605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443]}, [connected])
11:07:06.715 [reactor-http-nio-3] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] onStateChange(GET{uri=null, connection=PooledConnection{channel=[id: 0xa605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443]}}, [configured])
11:07:06.715 [reactor-http-nio-3] DEBUG reactor.netty.http.client.HttpClientConnect - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] Handler is being applied: {uri=https://vialivre-dev-image.servicebus.windows.net/$schemagroups/ImageSchemaRegistry/schemas/com.example.avro.Example?api-version=2020-09-01-preview, method=POST}
11:07:06.715 [reactor-http-nio-3] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] onStateChange(POST{uri=/$schemagroups/ImageSchemaRegistry/schemas/com.example.avro.Example, connection=PooledConnection{channel=[id: 0xa605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443]}}, [request_prepared])
11:07:06.715 [reactor-http-nio-3] DEBUG reactor.netty.ReactorNetty - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] Added decoder [azureWriteTimeoutHandler] at the end of the user pipeline, full pipeline: [reactor.left.sslHandler, reactor.left.httpCodec, azureWriteTimeoutHandler, reactor.right.reactiveBridge, DefaultChannelPipeline$TailContext#0]
11:07:06.723 [reactor-http-nio-3] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] onStateChange(POST{uri=/$schemagroups/ImageSchemaRegistry/schemas/com.example.avro.Example, connection=PooledConnection{channel=[id: 0xa605dfa6, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443]}}, [request_sent])
11:07:06.723 [reactor-http-nio-3] DEBUG reactor.netty.ReactorNetty - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] Removed handler: azureWriteTimeoutHandler, pipeline: DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
11:07:06.723 [reactor-http-nio-3] DEBUG reactor.netty.ReactorNetty - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] Added decoder [azureResponseTimeoutHandler] at the end of the user pipeline, full pipeline: [reactor.left.sslHandler, reactor.left.httpCodec, azureResponseTimeoutHandler, reactor.right.reactiveBridge, DefaultChannelPipeline$TailContext#0]
11:07:07.160 [reactor-http-nio-3] DEBUG reactor.netty.http.client.HttpClientOperations - [a605dfa6-1, L:/192.168.1.43:59873 - R:vialivre-dev-image.servicebus.windows.net/51.144.238.23:443] Received response (auto-read:false) : RESPONSE(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Transfer-Encoding: <filtered>
Content-Type: <filtered>
Location: <filtered>
Server: <filtered>
Serialization-Type: <filtered>
Schema-Id: <filtered>
Schema-Id-Location: <filtered>
Schema-Version: <filtered>
Schema-Versions-Location: <filtered>
Strict-Transport-Security: <filtered>
Date: <filtered>But if I recreate the schema in the portal (manually), the same request trying to get the same schema fails.
Due to a migration of azure-schemaregistry-kafka-avro of 1.0.0.beta.4 to 1.1.0-beta.1 we have detected that KafkaAvroDeserializer fails due to a problem with the Schema deserialization. To sum up, the first revision of the library registers the schema (auto.register.schema=true) and retrieve it as String (The JSON is inside the String) so when the new version of the library tries to retrieve it, it get as String and fail to parse the Schema.
Steps to reproduce with auto.register.schemas = true I have created a mock project located in my GitHub https://github.com/mgvinuesa/avro-eventhub-bug-steps with a test using the following versions:
Adding valid values for Schema (client, tenant...) in the test
AvroSerDesOldVersionTestyou are going to register the schema in the portal and deserialized it properly. Then the new schema is created and it is valid:Now, using the second project we are going to do the same thing, serialize and deserialize, in this case, the schema is not going to be created due to it already exists.
But in this case the schema deserialization fails. The error is:
The problem is that the received schema is a String, not a JSON, as you can see in the next debug screenshot:
You can see the scape characters to define the JSON inside the String. So the following line fails
Is there any release notes or capability to make this change backward compatible ? Due to currently all our schemas are created with this old library in the past, and now they are not compatible with the new one.