search

awslabs / kinesis-kafka-connector

kinesis-kafka-connector is connector based on Kafka Connect to publish messages to Amazon Kinesis streams or Amazon Kinesis Firehose.
Apache License 2.0
153 stars 91 forks source link

We occured connect-standalone error... plz help me... #4

Closed SungHyeon closed 7 years ago

SungHyeon commented 7 years ago

workers.properties

bootstrap.servers=172.0.XXX.XXX:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
#internal.value.converter=org.apache.kafka.connect.storage.StringConverter
#internal.key.converter=org.apache.kafka.connect.storage.StringConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter.schemas.enable=true
internal.value.converter.schemas.enable=true
offset.storage.file.filename=offset.log
schemas.enable=false

kinesis-streams-kafka-connector.properties

name=demo
connector.class=com.amazon.kinesis.kafka.AmazonKinesisSinkConnector
tasks.max=1
topics=log
region=ap-northeast-2
streamName=sastream
usePartitionAsHashKey=false
# Kinesis Producer Configuration - https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties
# All kinesis producer configuration have not been exposed
maxBufferedTime=1500
maxConnections=20
rateLimit=100
ttl=60000
metricsLevel=detailed
metricsGranuality=shard
metricsNameSpace=KafkaKinesisStreamsConnector
aggregration=true

command line ./connect-standalone.sh ../configs/worker.properties ../configs/kinesis-streams-kafka-connector.properties

error logs [2017-09-08 05:04:06,004] ERROR Failed to create job for ../configs/kinesis-streams-kafka-connector.properties (org.apache.kafka.connect.cli.ConnectStandalone:89) [2017-09-08 05:04:06,005] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:100) java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.amazon.kinesis.kafka.AmazonKinesisSinkConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'} at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79) at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:97) Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.amazon.kinesis.kafka.AmazonKinesisSinkConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='0.11.0.0', encodedVersion=0.11.0.0, type=source, typeName='source', location='classpath'} at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:165) at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:341) at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:240) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:157) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:94) [2017-09-08 05:04:06,008] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:65)

SungHyeon commented 7 years ago

I made the setting for $JAVA_HOME/lib/ext/amazon-kinesis-kafka-connecter-0.0.8-SNAPSHOT.jar

nehalmehta commented 7 years ago

Which version of Kafka are you using?

SungHyeon commented 7 years ago

My Kafka version here. kafka_version

nehalmehta commented 7 years ago

Hi @SungHyeon,

So latest version of Kafka Connect (including yours) requires to define ConfigDef function which was not required in earlier versions. Please build using latest Kinesis - Kafka Connector code and it should resolve your error. Latest code by @adrianfdzm includes ConfigDef. Let us know if you still face the issue.

Thanks, Nehal

SungHyeon commented 7 years ago

@nehalmehta Thank you. I will retry test for today.. And I would response for you.

nehalmehta commented 7 years ago

Closing due to lack of activity.