Cannot create topic - Githubissues

Fobhep commented 4 years ago

Hi - I am getting an error when I try to create a topic - not sure wether it's an actual "Kafka-Error" or more of a python import error

Expected Behavior

- hosts: localhost
  connection: local
  tasks:
    - name: Create list of zookeeper ips
      set_fact:
        zookeeper_ips: "{{ (zookeeper_ips | default([ ]) ) | union([item ~ ':2181']) }}"
      loop: "{{ groups['zookeeper'] }}"
    - name: Create list of kafka ips
      set_fact:
        kafka_ips: "{{ (kafka_ips | default([ ]) ) | union([item ~ ':2181']) }}"
      loop: "{{ groups['kafka_broker'] }}"

    - name: create topic
      kafka_lib:
        resource: "topic"
        api_version: "1.0.1"
        name: "test"
        partitions: 2
        replica_factor: 1
        options:
          retention.ms: 574930
          flush.ms: 12345
        state: "present"
        zookeeper: "{{ zookeeper_ips }}"
        bootstrap_servers: "{{ kafka_ips }}"

Expect a topic "test" to be created

Actual Behavior

TASK [create topic] ***********************************************************************************************************************************************************************************************
task path: /tmp/blub/topics.yml:15
Using module file /tmp/blub/library/kafka_lib.py
Pipelining is enabled.
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: mosfet
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "<stdin>", line 102, in <module>
  File "<stdin>", line 94, in _ansiballz_main
  File "<stdin>", line 40, in invoke_module
  File "/usr/lib/python2.7/runpy.py", line 188, in run_module
    fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/ansible_kafka_lib_payload_OW01V5/ansible_kafka_lib_payload.zip/ansible/modules/kafka_lib.py", line 21, in <module>
ImportError: No module named kafka.errors
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib/python2.7/runpy.py\", line 188, in run_module\n    fname, loader, pkg_name)\n  File \"/usr/lib/python2.7/runpy.py\", line 82, in _run_module_code\n    mod_name, mod_fname, mod_loader, pkg_name)\n  File \"/usr/lib/python2.7/runpy.py\", line 72, in _run_code\n    exec code in run_globals\n  File \"/tmp/ansible_kafka_lib_payload_OW01V5/ansible_kafka_lib_payload.zip/ansible/modules/kafka_lib.py\", line 21, in <module>\nImportError: No module named kafka.errors\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", 
    "rc": 1
}

Specifications

Library version:

Result of pip list command:

Package                 Version        
----------------------- ---------------
apturl                  0.5.2          
asn1crypto              0.24.0         
blinker                 1.4            
Brlapi                  0.6.7          
certifi                 2018.8.24      
chardet                 3.0.4          
command-not-found       0.3            
cryptography            2.6.1          
cupshelpers             1.0            
dbus-python             1.2.12         
defer                   1.0.6          
distro                  1.3.0          
distro-info             0.21ubuntu4    
entrypoints             0.3            
httplib2                0.11.3         
idna                    2.6            
kafka-python            1.4.4          
kazoo                   2.6.1          
keyring                 18.0.1         
keyrings.alt            3.1.1          
language-selector       0.1            
launchpadlib            1.10.7         
lazr.restfulclient      0.14.2         
lazr.uri                1.0.3          
louis                   3.10.0         
macaroonbakery          1.2.3          
netifaces               0.10.4         
oauthlib                2.1.0          
olefile                 0.46           
pbr                     5.1.1          
pexpect                 4.6.0          
Pillow                  6.1.0          
pip                     18.1           
protobuf                3.6.1          
psutil                  5.5.1          
pure-sasl               0.5.1          
pycairo                 1.16.2         
pycrypto                2.6.1          
pycups                  1.9.73         
Pygments                2.3.1          
PyGObject               3.34.0         
pyinotify               0.9.6          
PyJWT                   1.7.0          
pymacaroons             0.13.0         
PyNaCl                  1.3.0          
pyRFC3339               1.1            
python-apt              1.9.0+ubuntu1.3
python-dateutil         2.7.3          
python-debian           0.1.36         
python-pam              1.8.4          
python-xapp             1.6.0          
python-xlib             0.23           
pytz                    2019.2         
pyxdg                   0.25           
PyYAML                  5.1.2          
reportlab               3.5.23         
requests                2.21.0         
requests-unixsocket     0.1.5          
screen-resolution-extra 0.0.0          
SecretStorage           2.3.1          
setproctitle            1.1.10         
setuptools              41.1.0         
simplejson              3.16.0         
six                     1.12.0         
stevedore               1.31.0         
system-service          0.3            
systemd-python          234            
ubuntu-advantage-tools  19.5           
ubuntu-drivers-common   0.0.0          
ufw                     0.36           
unattended-upgrades     0.1            
urllib3                 1.24.1         
virtualenv              15.1.0         
virtualenv-clone        0.3.0          
virtualenvwrapper       4.8.4          
wadllib                 1.3.3          
wheel                   0.32.3         
xkit                    0.0.0

Kafka version: 2.4.1 (Confluent 5.4.1)
Python version: 2.7.17
OS: localhost: Ubuntu:19.10, Kafkaserver: Centos8

Fobhep commented 4 years ago

Ok - I could solve that issue by using python3. Using python2 I could not even get that working:

$ python
>> import kafka.errors
no such module kafka.errors

Fobhep commented 4 years ago

feel free to close this issue if that is ok for you

StephenSorriaux commented 4 years ago

Hello,

It seems like a dependency problem to me, are you using a virtualenv to start Ansible?

Fobhep commented 4 years ago

yes I do now. To get it running I had to install ansible also inside of the virtualenv besides the reqs of this repo in the virtualenv. Using my system-wide ansible-installation did not work.

StephenSorriaux commented 4 years ago

Seems ok to me then, thank you for your confirmation.

pgacek commented 3 years ago

@Fobhep could share how you managed to fix the issue? I think I got a similar problem. I'm trying to run the playbook inside of the docker container with the ansible and get the same error. I tried with a few Python versions, this is how my Dockerfile and ansible-playbook looks:

FROM alpine:3.7

RUN apk add --no-cache python3 git ansible \
&& python3 -m ensurepip \
&& pip3 install --upgrade pip setuptools \
&& rm -r /usr/lib/python*/ensurepip && \
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
rm -r /root/.cache

- name: Install pip modules
      become: yes
      pip:
        name: "{{ item }}"
      with_items:
        - kafka-python>=2.0.2
        - kazoo==2.6.1
        - pure-sasl==0.5.1

    - name: create topic
      kafka_topic:
        resource: "topic"
        api_version: "{{ kafka_api_version }}"
        name: "{{ item.name }}"
        partitions: "{{ item.partitions | default('8') }}"
        replica_factor: "{{ item.replica_factor | default('3') }}"
        state: "present"
        zookeeper: "{{ kafka_zookeeper_connection_string }}"
        bootstrap_servers: "{{ kafka_bootstrap_brokers }}"
        options:
            compression.type: "{{ item.options_compression_type | default('producer') }}"
            retention.ms: "{{ item.options_retention_ms | default('604800000') }}"
      with_items:
        - "{{ kafka_topics }}"

modules install correctly, but just after that a bunch of errors. I also installed the kafka package via the pip - didn't help

failed: [localhost] (item={u'name': u'aggregator-incoming-orders-events'}) => {"changed": false, "failed": true, "item": {"name": "aggregator-incoming-orders-events"}, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_e0r7Bv/ansible_module_kafka_topic.py\", line 15, in <module>\n    from ansible.module_utils.kafka_lib_topic import process_module_topic\n  File \"/tmp/ansible_e0r7Bv/ansible_modlib.zip/ansible/module_utils/kafka_lib_topic.py\", line 4, in <module>\nImportError: No module named kafka.errors\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}

Fobhep commented 3 years ago

As I can see you are installing ansible via package manager. I created a dedicated virtualenv and installed inside that ansible. I guess in your case where you don't seem to use a venv you could simply try to install ansible via pip3

pgacek commented 3 years ago

I changed the image and installed ansible via pip, as you suggested, but still get the same error message :(

FROM python:3.7.6-stretch

RUN pip install pip --upgrade
RUN pip install ansible

RUN apt-get update -y && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    sshpass

Fobhep commented 3 years ago

mmh - sorry then I don't really have other suggestions.

pgacek commented 3 years ago

Thanks a lot.

@StephenSorriaux maybe you would suggest something? :D

StephenSorriaux commented 3 years ago

Hey @pgacek

I think the become: yes may be the issue, since you are installing the libs for the root user. Do you still have the issue if you are removing the become: yes line or adding one to the call to create topic task?

    - name: Install pip modules
      become: yes
      pip:
        name: "{{ item }}"
      with_items:
        - kafka-python>=2.0.2
        - kazoo==2.6.1
        - pure-sasl==0.5.1

    - name: create topic
      kafka_topic:
        resource: "topic"
        api_version: "{{ kafka_api_version }}"
        name: "{{ item.name }}"
        partitions: "{{ item.partitions | default('8') }}"
        replica_factor: "{{ item.replica_factor | default('3') }}"
        state: "present"
        zookeeper: "{{ kafka_zookeeper_connection_string }}"
        bootstrap_servers: "{{ kafka_bootstrap_brokers }}"
        options:
            compression.type: "{{ item.options_compression_type | default('producer') }}"
            retention.ms: "{{ item.options_retention_ms | default('604800000') }}"
      with_items:
        - "{{ kafka_topics }}"

pgacek commented 3 years ago

actually, I'm getting another error - syntax

TASK [create topic] **************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SyntaxError: invalid syntax
failed: [localhost] (item={'name': 'aggregator-incoming-orders-events'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aggregator-incoming-orders-events"}, "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1620764755.0318098-98-39190393483405/AnsiballZ_kafka_topic.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1620764755.0318098-98-39190393483405/AnsiballZ_kafka_topic.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1620764755.0318098-98-39190393483405/AnsiballZ_kafka_topic.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.kafka_topic', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/local/lib/python3.7/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/local/lib/python3.7/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/local/lib/python3.7/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_kafka_topic_payload_pyvv4v_9/ansible_kafka_topic_payload.zip/ansible/modules/kafka_topic.py\", line 15, in <module>\n  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 967, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 668, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 638, in _load_backward_compatible\n  File \"/tmp/ansible_kafka_topic_payload_pyvv4v_9/ansible_kafka_topic_payload.zip/ansible/module_utils/kafka_lib_topic.py\", line 4, in <module>\n  File \"/usr/local/lib/python3.7/site-packages/kafka/__init__.py\", line 23, in <module>\n    from kafka.producer import KafkaProducer\n  File \"/usr/local/lib/python3.7/site-packages/kafka/producer/__init__.py\", line 4, in <module>\n    from .simple import SimpleProducer\n  File \"/usr/local/lib/python3.7/site-packages/kafka/producer/simple.py\", line 54\n    return '<SimpleProducer batch=%s>' % self.async\n                                                  ^\nSyntaxError: invalid syntax\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SyntaxError: invalid syntax
failed: [localhost] (item={'name': 'aggregator-incoming-orders-events.dlt', 'replica_factor': 2, 'partitions': 1}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aggregator-incoming-orders-events.dlt", "partitions": 1, "replica_factor": 2}, "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1620764755.3576074-98-148721332064075/AnsiballZ_kafka_topic.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1620764755.3576074-98-148721332064075/AnsiballZ_kafka_topic.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1620764755.3576074-98-148721332064075/AnsiballZ_kafka_topic.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.kafka_topic', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/local/lib/python3.7/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/local/lib/python3.7/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/local/lib/python3.7/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_kafka_topic_payload_s6t5jzfd/ansible_kafka_topic_payload.zip/ansible/modules/kafka_topic.py\", line 15, in <module>\n  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 967, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 668, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 638, in _load_backward_compatible\n  File \"/tmp/ansible_kafka_topic_payload_s6t5jzfd/ansible_kafka_topic_payload.zip/ansible/module_utils/kafka_lib_topic.py\", line 4, in <module>\n  File \"/usr/local/lib/python3.7/site-packages/kafka/__init__.py\", line 23, in <module>\n    from kafka.producer import KafkaProducer\n  File \"/usr/local/lib/python3.7/site-packages/kafka/producer/__init__.py\", line 4, in <module>\n    from .simple import SimpleProducer\n  File \"/usr/local/lib/python3.7/site-packages/kafka/producer/simple.py\", line 54\n    return '<SimpleProducer batch=%s>' % self.async\n                                                  ^\nSyntaxError: invalid syntax\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SyntaxError: invalid syntax

StephenSorriaux commented 3 years ago

I see, a SyntaxError is raised because of self.async in kafka-python, as async is a reserved keyword from Python 3.7 I think. But it seems like the version of kafka-python installed on the machine is not >=2.0.2 as this was changed in 1.4.3 release. Can you share the complete the playbook?

ryarnyah commented 3 years ago

@pgacek when you use python image you need to specify where python is located like in our tests: https://github.com/StephenSorriaux/ansible-kafka-admin/blob/master/molecule/default/molecule.yml#L238 If not you will can have 2 python version inside your image with one with kafka-python and the other no.

To be sure of which version your using, use /usr/local/bin/python -m pip install xxx when your calling pip.

StephenSorriaux / ansible-kafka-admin

Cannot create topic #54

Expected Behavior

Actual Behavior

Specifications