Update pulsar dependency to 2.10.3

datastax / cdc-apache-cassandra

Datastax CDC for Apache Cassandra

Apache License 2.0

35 stars 21 forks source link

Update pulsar dependency to 2.10.3 #144

Closed aymkhalil closed 1 year ago

aymkhalil commented 1 year ago

2.8 has performance reported issue, recommended to update to 2.10.x. Please note that the connector pulsar client will be vended by the environment, but this upgrade benefits the agent. It also unified the client version between the Agent and CLI tool.

Notes:

The maxPendingMessagesAcrossPartitions has been deprecated as of pulsar 2.10 (see PIP-120)

I couldn't upgrade to 2.11 because it required java 17. CDC need to compile with java 11 too. Sample build error:

bad class file: /home/runner/.gradle/caches/modules-2/files-2.1/org.apache.pulsar/pulsar-functions-api/2.11.0/6abd8cd6cf9b9a6ef15a7e53882577a897dcaf8a/pulsar-functions-api-2.11.0.jar(/org/apache/pulsar/functions/api/KVRecord.class)
class file has wrong version 61.0, should be 55.0

aymkhalil commented 1 year ago

@eolivelli

The maxPendingMessage is configured on the producer and has default value already
CNDB uses client 2.10.0 that doesn't have the fix for the memory limit. This patch uses 2.10.3 which has the fix
I added the configuration for memoryLimit that is disabled by default anyway.
maxPendingMessagesAcrossPartitions is deprecated and I think it is better to drop it as I don't expect future support for it

PTAL.

aymkhalil commented 1 year ago

Behavior of the JsonNode node returned in the NativeObject has changes. I have to tests failing now with:

java.lang.AssertionError: Wrong value for regular field double expected:<1.0> but was:<1>

With 2.8.3, I would get "1.0" when accessing node.numberValue() with DOUBEL schema. Now I get an "1" integer.

Need to troubleshoot further...

Update: Regression was introduced in 2.10.1. I captured the issue here https://github.com/apache/pulsar/issues/20092

In the mean time, I downgraded the client in the tests to 2.8.3 (like it was before). This Patch is still meaning full because it successfully update the client on the agent side to 2.10.3 and the connector client is only used in the integration tests.