datastax / cdc-apache-cassandra

Datastax CDC for Apache Cassandra
Apache License 2.0
35 stars 21 forks source link

start cdc agent failed #128

Open 115107496 opened 1 year ago

115107496 commented 1 year ago

I try to install CDC for Cassandra . when I start cassandra with agent, I encountered some errors,as bellow:

ERROR [CdcCommitlogProcessor:1] 2023-03-28 13:08:55,296 CommitLogReaderService.java:282 - Task segment=1679972621190 completed=true syncPosition=0 failed, retrying:
java.lang.NullPointerException: null
    at com.datastax.oss.cdc.agent.PulsarMutationSender.cqlToAvro(PulsarMutationSender.java:99)
    at com.datastax.oss.cdc.agent.PulsarMutationSender.cqlToAvro(PulsarMutationSender.java:34)
    at com.datastax.oss.cdc.agent.AbstractPulsarMutationSender.buildAvroKey(AbstractPulsarMutationSender.java:238)
    at com.datastax.oss.cdc.agent.AbstractPulsarMutationSender.sendMutationAsync(AbstractPulsarMutationSender.java:256)
    at com.datastax.oss.cdc.agent.CommitLogReadHandlerImpl.sendAsync(CommitLogReadHandlerImpl.java:468)
    at com.datastax.oss.cdc.agent.MutationMaker.createRecord(MutationMaker.java:32)
    at com.datastax.oss.cdc.agent.MutationMaker.createRecord(MutationMaker.java:24)
    at com.datastax.oss.cdc.agent.AbstractMutationMaker.insert(AbstractMutationMaker.java:28)
    at com.datastax.oss.cdc.agent.CommitLogReadHandlerImpl.handleRowModifications(CommitLogReadHandlerImpl.java:351)
    at com.datastax.oss.cdc.agent.CommitLogReadHandlerImpl.process(CommitLogReadHandlerImpl.java:310)
    at com.datastax.oss.cdc.agent.CommitLogReadHandlerImpl.handleMutation(CommitLogReadHandlerImpl.java:245)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:485)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:404)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:251)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:141)
    at com.datastax.oss.cdc.agent.CommitLogReaderServiceImpl$1.run(CommitLogReaderServiceImpl.java:64)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
    at java.lang.Thread.run(Thread.java:750)**_

my component information is as follows: cassandra3.11.14, pulsar 2.8.1, agent 1.0.5

is this a bug or config error?

thinks.

aymkhalil commented 1 year ago

It is possible that one of your primary key columns has an unsupported data type (are you able to share your table schema?) - you can cross check against this list: https://github.com/datastax/cdc-apache-cassandra/blob/ab6f5f96e1fe1604216175d01eedfea565862fb2/README.md#supported-data-types

aymkhalil commented 1 year ago

I'd also use a recent agent version - 1.0.5 is too old, maybe grab the llatest v2.2.3?

115107496 commented 1 year ago

It is possible that one of your primary key columns has an unsupported data type (are you able to share your table schema?) - you can cross check against this list: https://github.com/datastax/cdc-apache-cassandra/blob/ab6f5f96e1fe1604216175d01eedfea565862fb2/README.md#supported-data-types

here is the table schema,

create table properties_1636630056425668608
(
    "deviceId"   text,
    property     text,
    partition    bigint,
    "virtualId"  text,
    timestamp    bigint,
    "createTime" bigint,
    id           text,
    value        text,
    primary key (("deviceId", property, partition, "virtualId"), timestamp)
)
    with clustering order by (timestamp desc)
     and cdc = true;

I checked against the list you gave, and all column is supported.

115107496 commented 1 year ago

I'd also use a recent agent version - 1.0.5 is too old, maybe grab the llatest v2.2.3?

I will try the lastest version(v2.2.3), but I'd kown whether the v2.2.3 is match my env. thinks.