Closed ferrarimarco closed 6 years ago
cc @devbrand @StefanoSalsano @agil3b3ast
Another idea : from this comment on another kafka python client implementation, it is possible that the subscribe method is not supported in kafka broker 0.8.x
https://github.com/confluentinc/confluent-kafka-python/issues/146#issuecomment-285161275
We could try to use the assign method or to pass the topics as parameters to the KafkaConsumer constructor :
c=kafka.KafkaConsumer("scada", "env", bootstrap_servers='%s:%s'%(ip, port))
Do you have the environment setup to test these hypothesis ?
P.S: sorry for not using github, I don't have an account and don't want to create one on yet another website. Feel free to use my words to evolve the github issue.
Regards, Grégory.
From Grégory:
I tested on my side and can connect to the kafka instance with the python kafka client in version 1.3.2 (current version on my machine) and 1.3.5 (yours).
Both versions seems to work fine on my side. My test script :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import kafka
ip="159.100.240.40" # Update with "kafka" hostname
port="9092"
print("Kafka client version : %s" % (kafka.version.__version__,))
print("Connecting to %s:%s" % (ip, port))
c=kafka.KafkaConsumer(bootstrap_servers='%s:%s'%(ip, port))
c.subscribe(["scada", "env"])
for m in c:
print(m)
Also From Grégory:
Another idea : from this comment on another kafka python client implementation, it is possible that the subscribe method is not supported in kafka broker 0.8.x
https://github.com/confluentinc/confluent-kafka-python/issues/146#issuecomment-285161275
We could try to use the assign method or to pass the topics as parameters to the KafkaConsumer constructor :
c=kafka.KafkaConsumer("scada", "env", bootstrap_servers='%s:%s'%(ip, port))
@StefanoSalsano I think we can disable the failing test while waiting for a fix. I'll create a dedicated branch with the test enabled so we can experiment on a possible workaround.
By doing this we will have passing builds and it will be easier to check PRs. What do you think?
On 12/04/2018 11:09, Marco Ferrari wrote:
Thanks Grégory.
We are currently downloading kafka-idmef-converter 0.2.4 from https://yum.sixsq.com/scissor/yum as part of the Docker image build.
Are we going to find an updated version of the kafka-idmef-converter in that repo?
I pushed the updated converter and connector on the repository. I managed to test them and everything seems to work.
So upgrading kafka-idmef-converter to 0.2.5 should fix this!
Prerequisites
Description
When the
kafka-idmef-converter
andkafka-prelude-connector
start, I see some errors in the Kafka log. They cannot connect to kafka container (IP: 172.18.0.13).Host Environment
vagrant --version
): 2.0.2docker --version
): Docker version 18.02.0-ce, build fc4de44Output
Steps to Reproduce
Expected behavior:
kafka-idmef-converter
andkafka-prelude-connector
to connect tokafka
Actual behavior: Clients should be able to connect
Reproduces how often: always
Additional Information
We already ruled out incompatible client and server versions:
References
N/A