search

jcustenborder / kafka-connect-transform-common

Common Transforms for Kafka Connect.
Apache License 2.0
148 stars 53 forks source link

Unable to instantiate AdjustPrecisionAndScale: Plugin class must have a no-args constructor, and cannot be a non-static inner class #111

Open aonamrata opened 2 months ago

aonamrata commented 2 months ago

Hi,

I am trying to update our connectors and transform packages to resolve security vulnerabilities, so i updated my cp-kafka-connect-base:7.6.0 and jcustenborder/kafka-connect-transform-common:0.1.0.58 .

But now when i run it i get class must have a no-args constructor, and cannot be a non-static inner class error for following classes:

I am not using these transformers so i could ignore it. I am only using AddPrefix,changeTopicCase transformers. Is there a up coming release that will work with the new base image?

Full Docker file:

FROM confluentinc/cp-kafka-connect-base:7.6.0

USER root

# Run updates
RUN dnf -y upgrade && dnf clean all

# Update and install find for determing SSL configuration in entrypoint
RUN dnf -y install findutils jq

# Install Plugin

# Install Debezium Mysql connector from ConfluentHub
RUN confluent-hub install --no-prompt debezium/debezium-connector-mysql:1.9.7

RUN confluent-hub install --no-prompt jcustenborder/kafka-connect-transform-common:0.1.0.58

USER appuser

ENV CONNECTOR_NAME='debezium_mysql_source'
ENV CONNECT_REST_PORT=8083
ENV CONNECT_GROUP_UNIQUE_IDENTIFIER='namdebezium'

ENV CONNECT_KEY_CONVERTER='org.apache.kafka.connect.json.JsonConverter'
ENV CONNECT_VALUE_CONVERTER='org.apache.kafka.connect.json.JsonConverter'

ENV CONNECT_PLUGIN_PATH='/usr/share/java,/usr/share/confluent-hub-components/'
ENV CONNECT_SECURITY_PROTOCOL='SSL'
ENV CONNECT_CONFIG_PROVIDERS='file'
ENV CONNECT_CONFIG_PROVIDERS_FILE_CLASS='org.apache.kafka.common.config.provider.FileConfigProvider'
ENV CONNECT_CONSUMER_SECURITY_PROTOCOL='SSL'
ENV CONNECT_PRODUCER_SECURITY_PROTOCOL='SSL'
ENV CONNECT_CONSUMER_REQUEST_TIMEOUT_MS='20000'
ENV CONNECT_CONSUMER_RETRY_BACKOFF_MS='500'
ENV DATABASE_SERVER_ID='6000'
ENV DATABASE_TIMEZONE='America/New_York'
ENV DECIMAL_HANDLING_MODE='precise'
ENV Environment='dev'
ENV MAX_TASKS=1
ENV INCONSISTENT_SCHEMA_HANDLING_MODE='fail'
ENV CONNECT_REST_ADVERTISED_HOST_NAME='kafka-connect-debezium-mysql-source'

# Mitigate CVE-2021-44228
# https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/
ENV LOG4J_FORMAT_MSG_NO_LOOKUPS="true"

# Configure default log levels
ENV CONNECT_LOG4J_ROOT_LOGLEVEL="ERROR"
ENV CONNECT_LOG4J_LOGGERS="org.I0Itec.zkclient=ERROR,org.reflections=ERROR,org.eclipse.jetty=ERROR,kafka=ERROR,org.apache.kafka=ERROR,org.apache.zookeeper=ERROR,org.apache.kafka.connect.runtime.WorkerSourceTask=ERROR"

ENV CONNECT_GROUP_ID="kafka_connect_${CONNECTOR_NAME}_${CONNECT_GROUP_UNIQUE_IDENTIFIER}"
ENV CONNECT_CONFIG_STORAGE_TOPIC="_${CONNECT_GROUP_ID}-configs"
ENV CONNECT_OFFSET_STORAGE_TOPIC="_${CONNECT_GROUP_ID}-offsets"
ENV CONNECT_STATUS_STORAGE_TOPIC="_${CONNECT_GROUP_ID}-status"

# Connector topic config
ENV  DATABASE_HISTORY_KAFKA_TOPIC="_${DATABASE_SERVER_NAME}_history"

EXPOSE ${CONNECT_REST_PORT}

Full Error Log:

===> Check if Kafka is healthy ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/kafka/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Using log4j config /etc/cp-base-new/log4j.properties
===> Launching ...
===> Launching kafka-connect ...
[2024-04-17 08:24:17,691] ERROR Failed to discover Transformation in /usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-common: Unable to instantiate AdjustPrecisionAndScale: Plugin class must have a no-args constructor, and cannot be a non-static inner class (org.apache.kafka.connect.runtime.isolation.ReflectionScanner)
java.lang.NoSuchMethodException: com.github.jcustenborder.kafka.connect.transform.common.AdjustPrecisionAndScale.<init>()
        at java.base/java.lang.Class.getConstructor0(Class.java:3349)
        at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2553)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.versionFor(ReflectionScanner.java:73)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.getPluginDesc(ReflectionScanner.java:136)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.getTransformationPluginDesc(ReflectionScanner.java:106)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.scanPlugins(ReflectionScanner.java:91)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.scanUrlsAndAddPlugins(PluginScanner.java:79)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.discoverPlugins(PluginScanner.java:67)
        at org.apache.kafka.connect.runtime.isolation.Plugins.initLoaders(Plugins.java:90)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:74)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:64)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:121)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116)
[2024-04-17 08:24:17,694] ERROR Failed to discover Transformation in /usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-common: Unable to instantiate HeaderToField: Plugin class must have a no-args constructor, and cannot be a non-static inner class (org.apache.kafka.connect.runtime.isolation.ReflectionScanner)
java.lang.NoSuchMethodException: com.github.jcustenborder.kafka.connect.transform.common.HeaderToField.<init>()
        at java.base/java.lang.Class.getConstructor0(Class.java:3349)
        at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2553)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.versionFor(ReflectionScanner.java:73)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.getPluginDesc(ReflectionScanner.java:136)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.getTransformationPluginDesc(ReflectionScanner.java:106)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.scanPlugins(ReflectionScanner.java:91)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.scanUrlsAndAddPlugins(PluginScanner.java:79)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.discoverPlugins(PluginScanner.java:67)
        at org.apache.kafka.connect.runtime.isolation.Plugins.initLoaders(Plugins.java:90)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:74)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:64)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:121)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116)
[2024-04-17 08:24:17,695] ERROR Failed to discover Transformation in /usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-common: Unable to instantiate SetMaximumPrecision: Plugin class must have a no-args constructor, and cannot be a non-static inner class (org.apache.kafka.connect.runtime.isolation.ReflectionScanner)
java.lang.NoSuchMethodException: com.github.jcustenborder.kafka.connect.transform.common.SetMaximumPrecision.<init>()
        at java.base/java.lang.Class.getConstructor0(Class.java:3349)
        at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2553)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.versionFor(ReflectionScanner.java:73)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.getPluginDesc(ReflectionScanner.java:136)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.getTransformationPluginDesc(ReflectionScanner.java:106)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.scanPlugins(ReflectionScanner.java:91)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.scanUrlsAndAddPlugins(PluginScanner.java:79)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.discoverPlugins(PluginScanner.java:67)
        at org.apache.kafka.connect.runtime.isolation.Plugins.initLoaders(Plugins.java:90)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:74)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:64)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:121)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116)
[2024-04-17 08:24:18,023] ERROR Failed to discover Converter in /usr/share/confluent-hub-components/debezium-debezium-connector-mysql: Unable to instantiate CloudEventsConverter: Failed to statically initialize plugin class (org.apache.kafka.connect.runtime.isolation.ReflectionScanner)
java.lang.ExceptionInInitializerError
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.versionFor(ReflectionScanner.java:73)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.getPluginDesc(ReflectionScanner.java:136)
        at org.apache.kafka.connect.runtime.isolation.ReflectionScanner.scanPlugins(ReflectionScanner.java:89)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.scanUrlsAndAddPlugins(PluginScanner.java:79)
        at org.apache.kafka.connect.runtime.isolation.PluginScanner.discoverPlugins(PluginScanner.java:67)
        at org.apache.kafka.connect.runtime.isolation.Plugins.initLoaders(Plugins.java:90)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:74)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:64)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:121)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116)
Caused by: org.apache.kafka.connect.errors.DataException
        at io.debezium.converters.CloudEventsConverter.<clinit>(CloudEventsConverter.java:111)
rizvnn commented 1 week ago

Had a similar issue seting JAVA_OPTS ="-Xmx 2g -Xms 2g" on kafka connect container resolved it

aonamrata commented 6 days ago

Had a similar issue seting JAVA_OPTS ="-Xmx 2g -Xms 2g" on kafka connect container resolved it

Na, that did not work. I still get the same errors. I added this ENV to my docker compose file.