search

confluentinc / kafka-connect-hdfs

Kafka Connect HDFS connector
Other
9 stars 397 forks source link

Using kafka-connect-hdfs without confluent #175

Closed CBR09 closed 7 years ago

CBR09 commented 7 years ago

Hi, I want to write kafka topic to HDFS, I searched on google and found kafka-connect-hdfs, but it's related to confluent platform. Does kafka-connect-hdfs work without confluent platform?, I've kafka and hdfs which running in my environment. Thanks

cotedm commented 7 years ago

@CBR09 yes you can use this connector without using the Confluent platform in total. However, it's more straightforward to use the platform because everything comes in one bundle and you don't have to manage the installs separately. However, if you need to stay outside the platform, you'll need Apache Kafka 0.10 or higher and you will need to build the connector plugin jar yourself, installing it on the CLASSPATH properly: http://docs.confluent.io/3.1.1/connect/userguide.html#installing-connector-plugins

Then you'll need to think about converters and serialization. If you want to use Avro (recommended), we recommend installing the Schema Registry which includes all of the required converters and serializers. If you decide to use something else, you need to make sure you have a Formatter implementation for this connector that can be used to format the data being written to HDFS.

CBR09 commented 7 years ago

@cotedm Thank you for your reply, glad to hear that. I wonder is there any guide to build the kafka-connect-hdfs?, I can't find in docs.

cotedm commented 7 years ago

@CBR09 there's nothing in the docs currently. The command to build is mvn package to get you the required jars. You would then need to put the output of the build into your CLASSPATH. However, if you don't have experience doing this kind of things, I suggest you look into installing the packages provided by Confluent as everything is pre-packaged for you already. There is an open source version (left side here https://www.confluent.io/download-center/).

CBR09 commented 7 years ago

@cotedm thank you, I will give it a try

kanagha commented 6 years ago

Hi,

We are using kafka 0.11 version and we are looking into building kafka-connect-hdfs without using confluent platform. Investigating if 4.1.0-SNAPSHOT version would be compatible with our kafka binaries. Looking to see if there is a recommended way of pulling in older kafka-connect-hdfs required modules using correct git tags? We are investigating into 4.1.0-SNAPSHOT version. Please advise. Thanks!

rhauch commented 6 years ago

@kanagha in general you should be able to use a later version of a connector with an earlier version of Connect, unless that connector uses APIs that were introduced later. However, I'd recommend running a newer version of Connect and connectors (lots of bug fixes compared to 0.11), which you can use with older brokers.

kanagha commented 6 years ago

@rhauch, Thanks for the details. We are using kafka version 0.11.0.2. So, when I use the latest 5.0.0-SNAPSHOT version of kafka-connect-hdfs modules, compiling schema registry fails with the following error: While compiling schema-registry, I get the following error:

[INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/workspacename/kafka-connect/schema-registry/client/src/main/java/io/confluent/kafka/schemaregistry/client/security/basicauth/SaslBasicAuthCredentialProvider.java:[40,42] cannot find symbol symbol: method loadClientContext(java.util.Map<java.lang.String,java.lang.Object>) location: class org.apache.kafka.common.security.JaasContext [INFO] 1 error

Hence I'm trying to investigate with older versions of kafka-connect-hdfs modules. Also, I downloaded older binaries of confluent - https://www.confluent.io/previous-versions/ (ex: 4.0.1). All these binaries depend upon confluent.sh to start connects. Please advice if there is a recommended way to run kafka-connect-hdfs directly without using confluent platform?