search

StephenSorriaux / ansible-kafka-admin

Manage your topic's configuration (partitions, replication factor, parameters), ACLs, quotas, users and get stats, without any effort with this library. It does not use the Kafka scripts and does not require ssh connection to the remote broker.
Apache License 2.0
147 stars 46 forks source link

kafka 2.1 not work #41

Closed chris93111 closed 4 years ago

chris93111 commented 5 years ago

hello

i would like create topics with kafka 2.1 but is not work he could not connect to broker

Cannot determine a controller for your current Kafka server. Is your Kafka server running and available on '10.210.6.104' with security protocol 'PLAINTEXT'?"

StephenSorriaux commented 5 years ago

Hello,

I personally use this library on a daily basis with Kafka in version 2.1.1 so I don’t think it is related. How did you configure your Kafka broker (listeners, advertised address, etc.)?

chris93111 commented 5 years ago

all is default but i have try with

listeners=PLAINTEXT://:$myip9092 advertised.listeners=PLAINTEXT://$myip:9092

chris93111 commented 5 years ago

default is localhost and protocol PLAINTEXT

Zookeper and kakfka in same node

ryarnyah commented 5 years ago

Could you try adding:

import logging

log = logging.getLogger('kafka')
log.addHandler(logging.StreamHandler(sys.stdout))
log.setLevel(logging.INFO)

to get some logs?

imvld commented 4 years ago

hello. i have same problem kafka_2.12-2.3.0

ansible:

  - name: create acl
    kafka:
      resource: 'acl'
      api_version: "2.3.0"
      acl_resource_type: "topic"
      name: "*"
      acl_principal: "User:Anal"
      acl_operation: "all"
      acl_permission: "allow"
      state: "present"
      zookeeper: kafka.local:2181
      bootstrap_servers: kafka.local:9093
      security_protocol: 'SSL'
      ssl_check_hostname: true
      ssl_cafile: "{{ kafka.ssl.usersdir }}/ansible/CARoot.pem"
      ssl_certfile: "{{ kafka.users.ansible.cert }}"
      ssl_keyfile: "{{ kafka.users.ansible.key }}"
    tags: acl

ansibe

TASK [create acl] 
fatal: [kafka]: FAILED! => {"changed": false, "msg": "Cannot determine a controller for your current Kafka server. Is your Kafka server running and available on 'kafka.local:9093' with security protocol 'SSL'?"}

kafka:

listeners=SSL://kafka.local:9093
advertised.listeners=SSL://kafka.local:9093
advertised.host.name=kafka.local
ssl.keystore.location=/opt/kafka/ssl/kafka.server.keystore.jks
ssl.keystore.password=secret
ssl.key.password=secret
ssl.truststore.location=/opt/kafka/ssl/kafka.server.truststore.jks
ssl.truststore.password=secret
ssl.client.auth=required
listener.security.protocol.map=SSL:SSL
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=kafka.local;User:kafka;User:CN=ansible
allow.everyone.if.no.acl.found=false
security.inter.broker.protocol=SSL

netstat:

root@kafka:~# netstat -tulpan | grep -i listen|grep -P '(2181|9093)'
tcp6       0      0 10.254.242.241:9093     :::*                    LISTEN      23950/java      
tcp6       0      0 :::2181                 :::*                    LISTEN      21589/java  
root@kafka:~# grep -i endpoint /opt/kafka/kafka_2.12-2.3.0/logs/server.log|tail -1
[2019-10-01 14:03:30,093] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(kafka.local,9093,ListenerName(SSL),SSL)), czxid (broker epoch): 337 (kafka.zk.KafkaZkClient)

but with python i can connect to kafka

StephenSorriaux commented 4 years ago

Hi,

Which version of kafka-python are you using? Which version of the lib?

Can you please use the latest version of this lib (master or tag 0.8.0) and start your playbook in DEBUG mode: ANSIBLE_DEBUG=true ansible-playbook my-playbook.yml. Then, please copy and paste the line where kafka-python logs are present. This should be something like:

 13193 1569943505.41220: _low_level_execute_command() done: rc=0, stdout=Bootstrapping cluster metadata from [('localhost', 9092, <AddressFamily.AF_UNSPEC: 0>)]
<BrokerConnection node_id=bootstrap host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: connecting to localhost:9092 [('::1', 9092, 0, 0) IPv6]
<BrokerConnection node_id=bootstrap host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: Connection complete.
Bootstrap succeeded: found 1 brokers and 1 topics.
<BrokerConnection node_id=bootstrap host=localhost:9092 <connected> [IPv6 ('::1', 9092, 0, 0)]>: Closing connection.

{"changed": false, "msg": "topic 'test': nothing to do.", "invocation": {"module_args": {"resource": "topic", "api_version": "2.2.0", "name": "test", "partitions": 2, "replica_factor": 1, "options": {"retention.ms": 574930}, "state": "present", "zookeeper": "localhost:2181", "bootstrap_servers": "localhost:9092", "acl_resource_type": "topic", "acl_pattern_type": "literal", "acl_permission": "allow", "acl_host": "*", "zookeeper_auth_scheme": "digest", "zookeeper_auth_value": "", "zookeeper_ssl_check_hostname": true, "zookeeper_sleep_time": 5, "zookeeper_max_retries": 5, "security_protocol": "PLAINTEXT", "ssl_check_hostname": true, "sasl_mechanism": "PLAIN", "acl_principal": null, "acl_operation": null, "zookeeper_ssl_cafile": null, "zookeeper_ssl_certfile": null, "zookeeper_ssl_keyfile": null, "zookeeper_ssl_password": null, "ssl_cafile": null, "ssl_certfile": null, "ssl_keyfile": null, "ssl_password": null, "ssl_crlfile": null, "ssl_supported_protocols": null, "ssl_ciphers": null, "sasl_plain_username": null, "sasl_plain_password": null, "sasl_kerberos_service_name": null}}}
imvld commented 4 years ago

I used latest version of this lib(0.8.0)

root@kafka:~# pip show kafka-python|grep -i version
Version: 1.4.7
TASK [create acl] ********************************************************************************************************************************************************************************************************************
fatal: [kafka]: FAILED! => {"changed": false, "msg": "Cannot determine a controller for your current Kafka server. Is your Kafka server running and available on 'kafka.local:9093' with security protocol 'SSL'?"}
[kafka] TASK: create acl (debug)> p result._result
{'_ansible_no_log': False,
 '_ansible_parsed': True,
 'changed': False,
 u'failed': True,
 u'invocation': {u'module_args': {u'acl_host': u'*',
                                  u'acl_operation': u'all',
                                  u'acl_pattern_type': u'literal',
                                  u'acl_permission': u'allow',
                                  u'acl_principal': u'User:Anal',
                                  u'acl_resource_type': u'topic',
                                  u'api_version': u'2.3.0',
                                  u'bootstrap_servers': u'kafka.local:9093',
                                  u'name': u'*',
                                  u'options': None,
                                  u'partitions': 0,
                                  u'replica_factor': 0,
                                  u'resource': u'acl',
                                  u'sasl_kerberos_service_name': None,
                                  u'sasl_mechanism': u'PLAIN',
                                  u'sasl_plain_password': None,
                                  u'sasl_plain_username': None,
                                  u'security_protocol': u'SSL',
                                  u'ssl_cafile': u'/opt/kafka/ssl/users/ansible/CARoot.pem',
                                  u'ssl_certfile': u'/opt/kafka/ssl/users/ansible/ansible.kafka.cert',
                                  u'ssl_check_hostname': True,
                                  u'ssl_ciphers': None,
                                  u'ssl_crlfile': None,
                                  u'ssl_keyfile': u'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER',
                                  u'ssl_password': None,
                                  u'ssl_supported_protocols': None,
                                  u'state': u'present',
                                  u'zookeeper': u'kafka.local:2181',
                                  u'zookeeper_auth_scheme': u'digest',
                                  u'zookeeper_auth_value': u'',
                                  u'zookeeper_max_retries': 5,
                                  u'zookeeper_sleep_time': 5,
                                  u'zookeeper_ssl_cafile': None,
                                  u'zookeeper_ssl_certfile': None,
                                  u'zookeeper_ssl_check_hostname': True,
                                  u'zookeeper_ssl_keyfile': None,
                                  u'zookeeper_ssl_password': None}},
 u'msg': u"Cannot determine a controller for your current Kafka server. Is your Kafka server running and available on 'kafka.local:9093' with security protocol 'SSL'?"}
[kafka] TASK: create acl (debug)>
StephenSorriaux commented 4 years ago

Can you please retry with the 1.4.4 version of kafka-python as stated in the requirements.txt file?

imvld commented 4 years ago 
root@kafka:~# pip show kafka-python|grep -i version
Version: 1.4.4
  - pip:
      name: kafka-python
      version: 1.4.4
    tags: py

  - name: create acl
    kafka:
      resource: 'acl'
      api_version: "2.3.0"
      acl_resource_type: "topic"
      name: "*"
      acl_principal: "User:Anal"
      acl_operation: "all"
      acl_permission: "allow"
      state: "present"
      zookeeper: kafka.local:2181
      bootstrap_servers: kafka.local:9093
      security_protocol: 'SSL'
      ssl_check_hostname: true
      ssl_cafile: "{{ kafka.ssl.usersdir }}/ansible/CARoot.pem"
      ssl_certfile: "{{ kafka.users.ansible.cert }}"
      ssl_keyfile: "{{ kafka.users.ansible.key }}"
    tags: acl

thank you! it works )

root@kafka:~# /opt/kafka/kafka_2.12-2.3.0/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list|grep -i anal
    User:Anal has Allow permission for operations: All from hosts: *
StephenSorriaux commented 4 years ago

@imvld great, thanks for your feedback. @chris93111 can you check your versions on your side?

nkonopinski commented 4 years ago

Downgrading kafka-python from 1.4.6 to 1.4.4 did not completely resolve the issue for me:

failed: [host -> 127.0.0.1] (item=topic) => {"ansible_loop_var": "item", "changed": false, "item": "topic", "msg": "Error while initializing Zookeeper client : init() got unexpected keyword arguments: ['certfile', 'ca', 'keyfile_password', 'use_ssl', 'keyfile', 'verify_certs']. Is your Zookeeper server available and running on 'host1:2181,host2:2181,host3:2181'?"}

But then I upgraded kazoo from 2.5.0 to 2.6.1 and it is working now. Guess I should have read requirements.txt as well

StephenSorriaux commented 4 years ago

@nkonopinski thanks for your feedback.

I opened https://github.com/StephenSorriaux/ansible-kafka-admin/issues/43 that would help in the future