kafka-idmef-converter cannot connect to kafka container

ferrarimarco commented 6 years ago

Prerequisites

[x] Put an X between the brackets on this line if you have done all of the following:
- Checked that your issue isn't already filed: https://github.com/scissor-project/open-scissor/issues

Description

When the kafka-idmef-converter and kafka-prelude-connector start, I see some errors in the Kafka log. They cannot connect to kafka container (IP: 172.18.0.13).

Host Environment

Host operating system and version: Ubuntu 17.10
Vagrant version (output of vagrant --version): 2.0.2
Hypervisor version: Virtualbox 5.2.8
Docker version (output of docker --version): Docker version 18.02.0-ce, build fc4de44

Output

[2018-01-19 15:45:47,334] ERROR Closing socket for /172.18.0.13 because of error (kafka.network.Processor)
kafka.common.KafkaException: Wrong request type 18
    at kafka.api.RequestKeys$.deserializerForKey(RequestKeys.scala:64)
    at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:50)
    at kafka.network.Processor.read(SocketServer.scala:450)
    at kafka.network.Processor.run(SocketServer.scala:340)
    at java.lang.Thread.run(Thread.java:748)
[2018-01-19 15:45:47,439] ERROR Closing socket for /172.18.0.13 because of error (kafka.network.Processor)
kafka.common.KafkaException: Wrong request type 16
    at kafka.api.RequestKeys$.deserializerForKey(RequestKeys.scala:64)
    at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:50)
    at kafka.network.Processor.read(SocketServer.scala:450)
    at kafka.network.Processor.run(SocketServer.scala:340)
    at java.lang.Thread.run(Thread.java:748)

Steps to Reproduce

Start the test suite as suggested in the README

Expected behavior: kafka-idmef-converter and kafka-prelude-connector to connect to kafka

Actual behavior: Clients should be able to connect

Reproduces how often: always

Additional Information

We already ruled out incompatible client and server versions:

I tested on my side and can connect to the kafka instance with the python kafka client in version 1.3.2 (current version on my machine) and 1.3.5 (yours).

References

N/A

ferrarimarco commented 6 years ago

cc @devbrand @StefanoSalsano @agil3b3ast

ferrarimarco commented 6 years ago

Another idea : from this comment on another kafka python client implementation, it is possible that the subscribe method is not supported in kafka broker 0.8.x

https://github.com/confluentinc/confluent-kafka-python/issues/146#issuecomment-285161275

We could try to use the assign method or to pass the topics as parameters to the KafkaConsumer constructor :

c=kafka.KafkaConsumer("scada", "env", bootstrap_servers='%s:%s'%(ip, port))

Do you have the environment setup to test these hypothesis ?

P.S: sorry for not using github, I don't have an account and don't want to create one on yet another website. Feel free to use my words to evolve the github issue.

Regards, Grégory.

ferrarimarco commented 6 years ago

From Grégory:

I tested on my side and can connect to the kafka instance with the python kafka client in version 1.3.2 (current version on my machine) and 1.3.5 (yours).

Both versions seems to work fine on my side. My test script :

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import kafka

ip="159.100.240.40" # Update with "kafka" hostname
port="9092"
print("Kafka client version : %s" % (kafka.version.__version__,))
print("Connecting to %s:%s" % (ip, port))
c=kafka.KafkaConsumer(bootstrap_servers='%s:%s'%(ip, port))
c.subscribe(["scada", "env"])
for m in c:
    print(m)

ferrarimarco commented 6 years ago

Also From Grégory:

Another idea : from this comment on another kafka python client implementation, it is possible that the subscribe method is not supported in kafka broker 0.8.x

https://github.com/confluentinc/confluent-kafka-python/issues/146#issuecomment-285161275

We could try to use the assign method or to pass the topics as parameters to the KafkaConsumer constructor :

c=kafka.KafkaConsumer("scada", "env", bootstrap_servers='%s:%s'%(ip, port))

ferrarimarco commented 6 years ago

@StefanoSalsano I think we can disable the failing test while waiting for a fix. I'll create a dedicated branch with the test enabled so we can experiment on a possible workaround.

By doing this we will have passing builds and it will be easier to check PRs. What do you think?

ferrarimarco commented 6 years ago

On 12/04/2018 11:09, Marco Ferrari wrote:

Thanks Grégory.

We are currently downloading kafka-idmef-converter 0.2.4 from https://yum.sixsq.com/scissor/yum as part of the Docker image build.

Are we going to find an updated version of the kafka-idmef-converter in that repo?

I pushed the updated converter and connector on the repository. I managed to test them and everything seems to work.

So upgrading kafka-idmef-converter to 0.2.5 should fix this!

scissor-project / open-scissor