search

itadventurer / kafka-backup

Backup and Restore for Apache Kafka
Apache License 2.0
165 stars 46 forks source link

Cannot start backup: Missing required configuration "internal.key.converter" which has no default value #104

Open Damasker opened 4 years ago

Damasker commented 4 years ago

When I start backup

[root@ctrl kafka-backup]# /opt/kafka/bin/backup-standalone.sh --bootstrap-server kafka1:9092 --target-dir /backup/kafka/ --topics 'test-benchmark' --debug

/tmp/kafka-backup.zjmmPc/standalone.properties:

bootstrap.servers=kafka1:9092 key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter header.converter=org.apache.kafka.connect.converters.ByteArrayConverter key.converter.schemas.enable=false value.converter.schemas.enable=false offset.storage.file.filename=/tmp/kafka-backup.zjmmPc/kafka-backup.offsets offset.flush.interval.ms=10000 plugin.path=/opt/kafka/bin

/tmp/kafka-backup.zjmmPc/connector.properties:

name=backup-sink connector.class=de.azapps.kafkabackup.sink.BackupSinkConnector tasks.max=1 key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter header.converter=org.apache.kafka.connect.converters.ByteArrayConverter target.dir=/backup/kafka/ max.segment.size.bytes=1073741824 cluster.bootstrap.servers=kafka1:9092 topics:test-benchmark

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-elasticsearch/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-hdfs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-s3/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-storage-common/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration "internal.key.converter" which has no default value.
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:449)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:63)
        at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:163)
        at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.<init>(StandaloneConfig.java:43)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:69)
[root@ctrl kafka-backup]#

if I add to script these string:

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter

I got another error:

[root@ctrl kafka-backup]# /opt/kafka/bin/backup-standalone.sh --bootstrap-server kafka1:9092 --target-dir /backup/kafka/ --topics 'test-benchmark' --debug

/tmp/kafka-backup.vmz68Y/standalone.properties:

bootstrap.servers=kafka1:9092 key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter header.converter=org.apache.kafka.connect.converters.ByteArrayConverter key.converter.schemas.enable=false value.converter.schemas.enable=false internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter offset.storage.file.filename=/tmp/kafka-backup.vmz68Y/kafka-backup.offsets offset.flush.interval.ms=10000 plugin.path=/opt/kafka/bin

/tmp/kafka-backup.vmz68Y/connector.properties:

name=backup-sink connector.class=de.azapps.kafkabackup.sink.BackupSinkConnector tasks.max=1 key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter header.converter=org.apache.kafka.connect.converters.ByteArrayConverter target.dir=/backup/kafka/ max.segment.size.bytes=1073741824 cluster.bootstrap.servers=kafka1:9092 topics:test-benchmark

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-elasticsearch/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-hdfs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-s3/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-storage-common/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.connect.converters.ByteArrayConverter for configuration value.converter: Class org.apache.kafka.connect.converters.ByteArrayConverter could not be found.
        at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:703)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:446)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:63)
        at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:163)
        at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.<init>(StandaloneConfig.java:43)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:69)
[root@ctrl kafka-backup]#
itadventurer commented 4 years ago

Have you been able to resolve the issue? I think that may be a problem with the CLASSPATH…

Damasker commented 4 years ago

No, problem does not resolved.
Way of installing Kafka have matter? On test instance, kafka was installed from source. On production(where I have problem) installed from binary.

[root@ctrl mike]# export declare -x CLASSPATH=".:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/lib/tools.jar" declare -x GRADLE_HOME="/root/.sdkman/candidates/gradle/current" declare -x HISTSIZE="1000" declare -x HOME="/root" declare -x HOSTNAME="ctrl.nv.somehost.com" declare -x JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64" declare -x LANG="en_US.UTF-8" declare -x LESSOPEN="||/usr/bin/lesspipe.sh %s" declare -x LOGNAME="root" declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36:" declare -x MAIL="/var/spool/mail/mike" declare -x OLDPWD declare -x PATH="/root/.sdkman/candidates/gradle/current/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/bin" declare -x PWD="/home/mike" declare -x SDKMAN_CANDIDATES_DIR="/root/.sdkman/candidates" declare -x SDKMAN_CURRENT_API="https://api.sdkman.io/2" declare -x SDKMAN_DIR="/root/.sdkman" declare -x SDKMAN_LEGACY_API="https://api.sdkman.io/1" declare -x SDKMAN_PLATFORM="Linux64" declare -x SDKMAN_VERSION="5.5.9+231" declare -x SHELL="/bin/bash" declare -x SHLVL="1" declare -x SUDO_COMMAND="/bin/su" declare -x SUDO_GID="1004" declare -x SUDO_UID="1004" declare -x SUDO_USER="mike" declare -x TERM="xterm" declare -x USER="root" declare -x USERNAME="root" declare -x XDG_SESSION_ID="118707"

StupidToDie commented 4 years ago

i have the seem problem，did you have solve it

backup-standalone.sh --bootstrap-server localhost:9093 --target-dir /data/kafka_backup --topics 'first,test' Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration "internal.key.converter" which has no default value. at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:463) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:453) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:75) at org.apache.kafka.connect.runtime.WorkerConfig.(WorkerConfig.java:218) at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(StandaloneConfig.java:42) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:77)

Damasker commented 4 years ago

i have the seem problem，did you have solve it

backup-standalone.sh --bootstrap-server localhost:9093 --target-dir /data/kafka_backup --topics 'first,test' Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration "internal.key.converter" which has no default value. at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:463) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:453) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:75) at org.apache.kafka.connect.runtime.WorkerConfig.(WorkerConfig.java:218) at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(StandaloneConfig.java:42) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:77)

Hi! What way your kafka is installed?