[Question] ... Unable to get the data loaded to the topics eventhough I am able to see the kafkaconnector status running

[raviteja628]

kafkaconnector #kafkaconnect #strimzi kafka

Hi @scholzj @tombentley

I have deployed the kafka connect with the debezium oracle connector and trying to connect to oracle db to get the table data to the topics I have created the kafka-connector with the required configs and applied it on to the cluster namespace but the topics are not getting created . Any idea of workaround I need to do. just FYI I have also created the filestream source connector to load the file data into the topic , I tried to see the console consumer to check the data in the topic . I am struck at console with =LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) error. Attached is the screenshot for it

Can you help me with the probable workarounds to debug the issue

[scholzj]

Can you be more specific which topics are not being created? AFAIK Debezium creates its own topics it self. So it could be something about how you configured it? You also don't say anything about your configuration neither share any logs. So it is not obvious whether this could be for example authentication or authorization issue etc.

[raviteja628]

No topic is getting created as it is supposed to be auto created , do I need to configure any property to auto create topic. Please specify what config details you want specifically . I have deployed the configs in ref to https://strimzi.io/blog/2020/01/27/deploying-debezium-with-kafkaconnector-resource/ blog on strimzi.io . If it could be any of the authorization issue , can you pls guide me how can I trace back to get the logs concerned to it

[scholzj]

Well, you still didn't shared any logs or any other CRs. Nobody can really help you without that. Is the auto-creation enabled? Does everyone have the ACLs to use it?

[raviteja628]

can you help me with any reference in getting the required logs needed for you to know the what the issue. sorry that I am unable to figure out what logs needs to be shared with you .

[scholzj]

Basically, you should provide all the custom resources you are using to deploy Kafka, Connect etc. + logs from the related pods.

[raviteja628]

here you go scholz config file attached and logs of one of kafka pod that is kept repeating. hope you get some insight to help me

custom resources used here are

• kafkas
• kafkaconnects
• kafkaconnects2is
• kafkamirrormakers
• kafkausers
• kafkatopics
• kafkabridges
• kafkaconnectors

2020-10-19 16:34:25,318 DEBUG [Controller id=0] Topics not in preferred replica for broker 2 Map() (kafka.controller.KafkaController) [controller-event-thread] 2020-10-19 16:34:25,318 TRACE [Controller id=0] Leader imbalance ratio for broker 2 is 0.0 (kafka.controller.KafkaController) [controller-event-thread] 2020-10-19 16:34:25,318 DEBUG [Controller id=0] Topics not in preferred replica for broker 1 Map() (kafka.controller.KafkaController) [controller-event-thread] 2020-10-19 16:34:25,318 TRACE [Controller id=0] Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController) [controller-event-thread] 2020-10-19 16:34:25,318 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 Map() (kafka.controller.KafkaController) [controller-event-thread] 2020-10-19 16:34:25,318 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]

kafka-setup.txt

[scholzj]

The custom resources look good to me. The Kafka log messages do not mean anything wrong AFAIK. That is just a usual chatter in the Kafka logs.

[raviteja628]

Can we get logs of kafka-connect pod by anyway so that we might get some insights. I am not able to run a simple file streamsource kafkaconnector , eventhough the status gives it look running state in the kubectl describe kctr without any error msg but I am unable to see the msgs in the topic. I am not able to figure out what would go wrong. Any kind of basic troubleshooting help on kafka-connect would be much appreciated

[scholzj]

Every application is logging within its pod. So you should have all the logs collected inside your cluster or at least at a best effort available with kubectl logs

[raviteja628]

Hi @scholzj @tombentley

I have successfully able to run the debezium connector after adding the instaclient path in kafkaconnect spec as suggested in issue #2507 to resolve ocijdbc19 (not found in java.library.path) exception but I have an issue here , for the first time it is able to create table topic and suddenly again throwing the same here as above and stopped the connector. Any heads up here why is it happening

Note : there is also no data pushed into the topic and could only see the topic created Below are the details of connector spec yaml , connector configs with the error , connect logs and topic created.

connector spec yaml: kafka_connect_jdbc_oracle.txt

Created topic: schema-changes.dashboard-metadata

connector configs with the error :

Kafka-connect logs:

connect_log.txt

[scholzj]

If you check how your KafkaConnect resource looks like, it differs from what is described in https://github.com/strimzi/strimzi-kafka-operator/issues/2507#issuecomment-589971572 ... so you did not set any of the variables and that is why it is probably failing.

[raviteja628]

I have set the kafkaconnect with the below variables @scholzj as suggested in #2507 issue .

env:

• name: KAFKA_OPTS value: "-Djava.library.path=/opt/kafka/plugins/debezium"
• name: LD_LIBRARY_PATH value: "/opt/kafka/plugins/debezium"

Anything I have missed here.

[scholzj]

Please compare the YAML structure in my comment in #2507 with the KAfkaConnect CR you have attached above. They have very different structure.

[raviteja628]

Thank you so much @scholzj. It worked with the KAFKA_OPTS option and now having a new error now with this .

Caused by: java.lang.UnsatisfiedLinkError: /opt/kafka/plugins/debezium/libocijdbc19.so: libclntsh.so.19.1: cannot open shared object file: No such file or directory

I have both the files in the directory , still this error persists.

any dependency need to be needed in the image to read this files!!

[scholzj]

I'm afraid I have no clue about this - I know only little about Debezium and nothing about the Oracle binaries it is using. Sorry.

[ruettere]

@raviteja628 Hi, my approach is to set these variables in the dockerfile which is used then to build the connect image:

FROM strimzi/kafka:0.19.0-kafka-2.5.0 USER root:root ENV CONNECT_PLUGIN_PATH=/opt/kafka/plugins ENV PATH=$PATH:/opt/kafka/plugins:/opt/kafka/libs ENV CLASSPATH=/opt/kafka/libs/ojdbc8.jar ENV INSTANT_CLIENT_DIR=/instant_client/ ENV LD_LIBRARY_PATH=/instant_client

RUN mkdir -p /opt/kafka/plugins/debezium-connector-oracle RUN mkdir -p /instant_client/

COPY ./ora_client/instantclient_12_2/* $INSTANT_CLIENT_DIR COPY ./ora_client/instantclient_12_2/ojdbc8.jar /opt/kafka/libs/ COPY ./ora_client/instantclient_12_2/xstreams.jar /opt/kafka/libs/ COPY ./plugins/kafka-connect-jdbc-5.5.1.jar /opt/kafka/libs/

COPY ./plugins/debezium-connector-oracle/ /opt/kafka/plugins/debezium-connector-oracle/ USER 1001

What does yours look like (raviteja628/strimzi_kafka_oracle:5.0)? Is your connector running in the meantime?

[scholzj]

@raviteja628 Does this work for you now? Can we close this issue?