Contribute to Apache?

[mumrah]

Is there any plan to contribute this to Apache Camel? I think Kafka aligns well with their design and this components would be well received.

[Giwi]

Hi, why not. I don't know how to do that, but, if you have any clue, I'll be happy :)

[mumrah]

Typically the process is to open a JIRA ticket with a subject like "Adds Kafka component for Camel". I've pulled some of your code into an internal project at work to make it work for our use case. I plan to update things for the upcoming Kafka 0.8 release and add some features. Unless you'd like to handle it, I'd eventually like to contribute it to Apache (with credit due where credit is due, of course).

[Giwi]

You can fork me and submit a pull request. I will open a Jira ticket in a couple of days. Well, I enjoy to see that my work is useful :)

[bellwethr]

Any progress here? We're potentially interested in using this as well.

[Giwi]

Hi, I've started the integration of this component with the official dev of camel, but it's a tough work. Anyway, you can use this component right now from my repo, it works well (we use it in my company). Any feedback will be nice.

[mumrah]

@Giwi I've been using a modified version of this library for a few weeks now in development. So far, so good.

One significant change I made was to use generic Properties in KafkaEndpoint rather than enumerating all the properties. E.g., in KafkaEndpoint I have three instance variables: String topicName, Properties props, int concurrentConsumers

I've also reworked things a bit to make it Spring friendly (since that's how we configure Camel).

For example, here is how I configure a producer endpoint:

  <bean id="kafka-in" class="com.mycompany.component.kafka.KafkaEndpoint">
    <property name="topicName" value="documents.raw"/>
    <property name="props">
      <props>
        <prop key="zk.connect">localhost:2181/kafka</prop>
        <prop key="zk.connectiontimeout.ms">60000</prop>
        <prop key="zk.sessiontimeout.ms">120000</prop>
      </props>
    </property>
  </bean>

And the consumer endpoint:

  <bean id="kafka-out" class="com.mycompany.component.kafka.KafkaEndpoint">
    <property name="topicName" value="documents.raw"/>
    <property name="props">
      <props>
        <prop key="zk.connect">localhost:2181/kafka</prop>
        <prop key="zk.connectiontimeout.ms">60000</prop>
        <prop key="zk.sessiontimeout.ms">120000</prop>
        <prop key="groupid">camel</prop>
      </props>
    </property>
    <property name="concurrentConsumers" value="4"/>
  </bean>

I did a wholesale copy of your code into our repo, so formatting a pull request might be tricky. I also want to upgrade to Kafka 0.8 once that is released.

[Giwi]

Ok, great job! You can make a pull-request as soon as you want. I wait for the next release of Kafka for modifying the component for accepting any version of Kafka.

[bellwethr]

@mumrah -- anyway to access your fork? We're using Kafka 0.8 beta right now and I'm investigating Spring integration of Kafka with CXF; CXF has support for Camel-based connections, which lead me here...