ImportError: No module named kafka.client

[imalik8088]

Hi @StephenSorriaux great work that you have done!!
I'm trying to test and include the your module in my setup and I have some issues:

Stacktrace:

$ ansible-playbook deploy-topics.yml -vvv
ansible-playbook 2.6.4
  config file = /Users/imalik/dir/dir/ansible.cfg
  configured module search path = [u'/Users/imalik/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.6.4/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.15 (default, Sep 18 2018, 20:16:18) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
Using /Users/USER/dir/dir/ansible.cfg as config file
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAYBOOK: deploy-topics.yml ************************************************************************************************************
1 plays in deploy-topics.yml

PLAY [127.0.0.1] ***********************************************************************************************************************
META: ran handlers

TASK [kafka_lib : create topic] ********************************************************************************************************
task path: /Users/imalik/code/dir/dir/roles/kafka_lib/tasks/main.yml:2
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: imalik
<127.0.0.1> EXEC /bin/sh -c 'echo ~imalik && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/imalik/.ansible/tmp/ansible-tmp-1541780162.6-66828776799411 `" && echo ansible-tmp-1541780162.6-66828776799411="` echo /Users/imalik/.ansible/tmp/ansible-tmp-1541780162.6-66828776799411 `" ) && sleep 0'
Using module file /Users/imalik/code/dir/dir/roles/kafka_lib/library/kafka_lib.py
<127.0.0.1> PUT /Users/imalik/.ansible/tmp/ansible-local-53388kHbcXW/tmp4ZMbAH TO /Users/imalik/.ansible/tmp/ansible-tmp-1541780162.6-66828776799411/kafka_lib.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/imalik/.ansible/tmp/ansible-tmp-1541780162.6-66828776799411/ /Users/imalik/.ansible/tmp/ansible-tmp-1541780162.6-66828776799411/kafka_lib.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /Users/imalik/.ansible/tmp/ansible-tmp-1541780162.6-66828776799411/kafka_lib.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/imalik/.ansible/tmp/ansible-tmp-1541780162.6-66828776799411/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_ae_9UZ/ansible_module_kafka_lib.py", line 199, in <module>
    from kafka.client import KafkaClient
ImportError: No module named kafka.client

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_ae_9UZ/ansible_module_kafka_lib.py\", line 199, in <module>\n    from kafka.client import KafkaClient\nImportError: No module named kafka.client\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}

PLAY RECAP *****************************************************************************************************************************
127.0.0.1                  : ok=0    changed=0    unreachable=0    failed=1

I've your module in a requirements.yml and run this command ansible-galaxy install -r requirements.yml --roles-path roles cd to that role and pip install -r requirements.txt the libs were installed without an issue.

I have added a tasks and main.yml in the kafka_lib folder an used your example basically creation of a simple topic and it fails with No module xxx found. Do you have any idea or hint to look for?

[StephenSorriaux]

Hi @imalik8088, thanks for your message!

It seems like the kafka-python lib cannot be reached, did you pip install -r requirements.txt from inside a virtualenv? On your host, can you do a python -c 'from kafka.client import KafkaClient' without any problem?

[imalik8088]

it seems that my laptop and python are not working correctly. The inline command with python -c was also not able to find the lib. I wish could understand python better 💭

I have installed on my local machine python@2 via homebrew and my python --version is 2.7.5 have also tried todo pip3 install -r requirements.txt

[imalik8088]

ok think I found the solution :-) python -m pip install -r requirements.txt for my mac and homebrew installtion. but now the next issue

The full traceback is:
Traceback (most recent call last):
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py", line 363, in <module>
    class CreatePartitionsRequest_v0(Request):
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py", line 366, in CreatePartitionsRequest_v0
    RESPONSE_TYPE = CreatePartitionsResponse_v0
NameError: name 'CreatePartitionsResponse_v0' is not defined

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py\", line 363, in <module>\n    class CreatePartitionsRequest_v0(Request):\n  File \"/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py\", line 366, in CreatePartitionsRequest_v0\n    RESPONSE_TYPE = CreatePartitionsResponse_v0\nNameError: name 'CreatePartitionsResponse_v0' is not defined\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}

https://github.com/StephenSorriaux/ansible-kafka-admin/blob/master/library/kafka_lib.py#L366 I think the issue is here. The CreatePartitionsResponse_v0 is not defined.

[StephenSorriaux]

Great. Are you using the 1.4.2 version of kafka-python? The CreatePartitionsResponse_v0 is taken from this library and imported via from kafka.protocol.admin import *.

[imalik8088]

Wow thx for the support! But unfortunately it bumped again

The full traceback is:
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_aJu4GQ/ansible_module_kafka_lib.py", line 758, in main
    manager = KafkaManager(module=module, bootstrap_servers=bootstrap_servers, security_protocol=security_protocol, api_version=api_version, ssl_check_hostname=ssl_check_hostname, ssl_cafile=ssl_files['cafile']['path'], ssl_certfile=ssl_files['certfile']['path'], ssl_keyfile=ssl_files['keyfile']['path'], ssl_password=ssl_password, ssl_crlfile=ssl_files['crlfile']['path'], sasl_mechanism=sasl_mechanism, sasl_plain_username=sasl_plain_username, sasl_plain_password=sasl_plain_password, sasl_kerberos_service_name=sasl_kerberos_service_name)
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_aJu4GQ/ansible_module_kafka_lib.py", line 382, in __init__
    self.client = KafkaClient(**configs)
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/kafka/client_async.py", line 192, in __init__
    self.config['api_version'], str(self.API_VERSIONS)))

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "1.0.1",
            "bootstrap_servers": "localhost:9092",
            "name": "test",
            "options": {
                "flush.ms": 12345,
                "retention.ms": 574930
            },
            "partitions": 2,
            "replica_factor": 1,
            "resource": "topic",
            "sasl_kerberos_service_name": null,
            "sasl_mechanism": "PLAIN",
            "sasl_plain_password": null,
            "sasl_plain_username": null,
            "security_protocol": "PLAINTEXT",
            "ssl_cafile": null,
            "ssl_certfile": null,
            "ssl_check_hostname": true,
            "ssl_crlfile": null,
            "ssl_keyfile": null,
            "ssl_password": null,
            "state": "present",
            "zookeeper": "localhost:2181",
            "zookeeper_auth_scheme": "digest",
            "zookeeper_auth_value": ""
        }
    },
    "msg": "Error while initializing Kafka client : api_version [('1', '0', '1')] must be one of: [(0, 10, 1), (0, 10, 0), (0, 10), (0, 9), (0, 8, 2), (0, 8, 1), (0, 8, 0)] "
}

I'm using confluent-4.1.1, means Kafka 1.1.x https://docs.confluent.io/current/installation/versions-interoperability.html#cp-versions

[StephenSorriaux]

I think you are not using the 1.4.2 version of kafka-python since the api_version check was removed from the 1.4.2 version (https://github.com/dpkp/kafka-python/commit/22e3f75e92a791b3a42cac2a87b19ec33a4ca351)

[imalik8088]

Had to reinstall the deps but i thinks it looks good. Now I have to solve somehow the connection issue.

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "1.0.1",
            "bootstrap_servers": "localhost:9092",
            "name": "test",
            "options": {
                "flush.ms": 12345,
                "retention.ms": 574930
            },
            "partitions": 2,
            "replica_factor": 1,
            "resource": "topic",
            "sasl_kerberos_service_name": null,
            "sasl_mechanism": "PLAIN",
            "sasl_plain_password": null,
            "sasl_plain_username": null,
            "security_protocol": "PLAINTEXT",
            "ssl_cafile": null,
            "ssl_certfile": null,
            "ssl_check_hostname": true,
            "ssl_crlfile": null,
            "ssl_keyfile": null,
            "ssl_password": null,
            "state": "present",
            "zookeeper": "localhost:2181",
            "zookeeper_auth_scheme": "digest",
            "zookeeper_auth_value": ""
        }
    },
    "msg": "Connection is not ready, please check your client and server configurations."
}

[StephenSorriaux]

OK, have you got any logs from your Kafka server? Can you give its configuration?

[imalik8088]

Strage when I run the playbook they no logs from from Kafka or ZK are shown up. Just for testing purpose I have used curl with localhost:2181 and the ZK logs appeared 🤔

I have upload now the server and the Ansible project:

I'm running the Kafka Cluster with with docker. If you want to try it out. Just clone the following repo https://github.com/imalik8088/kafka-on-demand-docker and docker-compose -f my-stack.yml up

And the Ansible stuff is here: https://github.com/imalik8088/ansible-kafka-topics-example and there I have run ansible-playbook deploy-topics.yml

[StephenSorriaux]

Thanks for the code, it was very helpful for debugging.

The problem comes from the Kafka broker configuration, it currently is advertising to PLAINTEXT://kafka:9092 which is not reachable from outside the container unless you add an entry to your /etc/hosts file with the current IP of the Kafka container. This is not very easy to use since if the container's IP change, you will need to update your /etc/hosts... To avoid this, here the configuration of your Kafka broker for the my-stack.yml file:

  kafka:
    image: confluentinc/cp-kafka:${CONFLUENT_VERSION}
    container_name: kafka_kodd
    hostname: kafka
    depends_on:
      - zookeeper
    ports:
      - "9094:9094"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092,OUTSIDE://localhost:9094"
      KAFKA_LISTENERS: "PLAINTEXT://:9092,OUTSIDE://:9094"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_LOG4J_LOGGERS: "kafka.controller=DEBUG,kafka.producer.async.DefaultEventHandler=DEBUG,state.change.logger=DEBUG"
      KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_DELETE_TOPIC_ENABLE: "true"

With this configuration, your Kafka broker will be reachable from outside the container via PLAINTEXT://localhost:9094. Change the port accordingly inside your tasks/main.yml and please use the master branch of my library. Then you will be good to go: (ansible)

$ ansible-playbook deploy-topics.yml -vv
ansible-playbook 2.5.2
  config file = /home/stephen/dev/tmp/ansible-kafka-topics-example/ansible.cfg
  configured module search path = [u'/home/stephen/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.15rc1 (default, Apr 15 2018, 21:51:34) [GCC 7.3.0]
Using /home/stephen/dev/tmp/ansible-kafka-topics-example/ansible.cfg as config file
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match
'all'

PLAYBOOK: deploy-topics.yml *******************************************************************************************
1 plays in deploy-topics.yml

PLAY [127.0.0.1] ******************************************************************************************************
META: ran handlers

TASK [kafka_lib : create topic] ***************************************************************************************
task path: /home/stephen/dev/tmp/ansible-kafka-topics-example/roles/kafka_lib/tasks/main.yml:1
changed: [127.0.0.1] => {"changed": true, "msg": "topic 'test1': successfully created."}
META: ran handlers
META: ran handlers

PLAY RECAP ************************************************************************************************************
127.0.0.1                  : ok=1    changed=1    unreachable=0    failed=0

(kafka)

root@kafka:/# kafka-topics --describe --zookeeper zookeeper:2181
Topic:__confluent.support.metrics   PartitionCount:1    ReplicationFactor:1 Configs:retention.ms=31536000000
    Topic: __confluent.support.metrics  Partition: 0    Leader: 1   Replicas: 1 Isr: 1
Topic:_schemas  PartitionCount:1    ReplicationFactor:1 Configs:cleanup.policy=compact
    Topic: _schemas Partition: 0    Leader: 1   Replicas: 1 Isr: 1
Topic:test1 PartitionCount:2    ReplicationFactor:1 Configs:retention.ms=574930,flush.ms=12345
    Topic: test1    Partition: 0    Leader: 1   Replicas: 1 Isr: 1
    Topic: test1    Partition: 1    Leader: 1   Replicas: 1 Isr: 1

[imalik8088]

Yeah buddy thx very much for the help and once again great work! Would love to provide a more sophisticated example project, after my short holiday. Examining currently your Ansible module at work were the customer has some requirements or an old way to manage there topics with custom gradle, ruby and bash scripts 🤕 🤒 lets see if I can make world a better place to live 😄

[StephenSorriaux]

Sure, I’m happy to help. Feel free to submit any bugs or new features in the future.

I’m now closing this issue since the underlying problem is now solved.