Get unspecified Error Message on module kafka_info for topics

david-freistrom commented 2 years ago

Expected Behavior

Get a List of all topics inside the kafka cluster.

Actual Behavior

Get Error: fatal: [localhost]: FAILED! => {"changed": false, "msg": "Seomthing went wrong: 'min.insync.replicas' "}

The Error maybe was thrown here: https://github.com/StephenSorriaux/ansible-kafka-admin/blob/547ff8c38b7383bdc3f4fafb1d303d28d1c484ae/library/kafka_info.py#L90

By the way: You have an typo here too

Play to Reproduce the Problem

0) A kafka cluster deployed via Helm

helm upgrade -i -n kafka kafka bitnami/kafka --set auth.tls.type=pem --set deleteTopicEnable=true --set auth.clientProtocol=tls --set auth.interBrokerProtocol=tls --set auth.tls.existingSecrets[0]=kafka-broker-tls --set replicaCount=1 --set authorizerClassName=kafka.security.authorizer.AclAuthorizer

1) At least one topic needs to be exist

- name: "TopicClaim: Find an existing Kafka Topic for that Claim"
  StephenSorriaux.ansible_kafka_admin.kafka_info:
    resource: "topic"
    bootstrap_servers: "{{ bootstrap_servers }}"
    security_protocol: 'SSL'
    ssl_cafile: "{{ ssl_cafile }}"
    ssl_certfile: "{{ ssl_certfile }}"
    ssl_keyfile: "{{ ssl_keyfile }}"
  vars:
    bootstrap_servers: "{{ lookup('ENV', 'KAFKA_BOOTSTRAP_SERVERS') }}"
    ssl_cafile: "{{ lookup('ENV', 'KAFKA_CA_CERT') }}"
    ssl_certfile: "{{ lookup('ENV', 'KAFKA_CLIENT_CERT') }}"
    ssl_keyfile: "{{ lookup('ENV', 'KAFKA_CLIENT_KEY') }}"
  register: kafka_topics

Logs from the play with Ansible in debug mode

ANSIBLE_DEBUG=true ansible-playbook my-awesome-playbook.yml

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Seomthing went wrong: 'min.insync.replicas' "}

...task path: /opt/ansible/roles/topic/tasks/main.yml:2\nincluded: /opt/ansible/roles/topic/tasks/deploy.yml for localhost\n\r\nTASK [topic : Topic: Find an existing Kafka Topic for that Claim] **************\r\ntask path: /opt/ansible/roles/topic/tasks/deploy.yml:2\n[WARNING]: Module invocation had junk after the JSON data: <BrokerConnection\r\nnode_id=0 host=kafka-0.kafka-headless.kafka.svc.cluster.local:9092 <connected>\r\n[IPv4 ('10.245.0.9', 9092)]>: Closing connection.\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Seomthing went wrong: 'min.insync.replicas' \"}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 \r\n\n","job":"8768220180229646268","name":"topic1","namespace":"ipl-operator-system","error":"exit status 2"}

The full traceback is:
File "/tmp/ansible_StephenSorriaux.ansible_kafka_admin.kafka_info_payload_c9e18gfv/ansible_StephenSorriaux.ansible_kafka_admin.kafka_info_payload.zip/ansible_collections/StephenSorriaux/ansible_kafka_admin/plugins/modules/kafka_info.py", line 81, in main
File "/tmp/ansible_StephenSorriaux.ansible_kafka_admin.kafka_info_payload_c9e18gfv/ansible_StephenSorriaux.ansible_kafka_admin.kafka_info_payload.zip/ansible_collections/StephenSorriaux/ansible_kafka_admin/plugins/module_utils/kafka_manager.py", line 1379, in get_resource return self.resource_to_func[resource]()
File "/tmp/ansible_StephenSorriaux.ansible_kafka_admin.kafka_info_payload_c9e18gfv/ansible_StephenSorriaux.ansible_kafka_admin.kafka_info_payload.zip/ansible_collections/StephenSorriaux/ansible_kafka_admin/plugins/module_utils/kafka_manager.py", line 1195, in get_topics_resource
min_isr = int(topic_config['min.insync.replicas'])
fatal: [localhost]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"api_version": "7.0.1",
"bootstrap_servers": "kafka.kafka.svc:9092",
"connections_max_idle_ms": 540000,
"request_timeout_ms": 60000,
"resource": "topic",
"sasl_kerberos_service_name": null,
"sasl_mechanism": "PLAIN",
"sasl_plain_password": null,
"sasl_plain_username": null,
"security_protocol": "SSL",
"ssl_cafile": "-----BEGIN CERTIFICATE-----\nMIIBhDCCASugAwIBAgIRAJzCZyq8UCOzTAMp71SoE8YwCgYIKoZIzj0EAwIwIjEg\nMB4GA1UEAxMXY2x1c3Rlci1pbnRlcm1lZGlhdGUtY2EwHhcNMjIwMTE3MDY0MTQx\nWhcNMjIwNDE3MDY0MTQxWjAiMSAwHgYDVQQDExdjbHVzdGVyLWludGVybWVkaWF0\nZS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJyVv/tULyk2247b20HOZd/2\n44mqejLiCY7MEHxISyeGfzVYhNvlUl+SI69wyGEylKSBRHEX0jt9XBAUQJ+EiCij\nQjBAMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRe\nM9YQsPK8Gy/ioQccN3XcTTtkEDAKBggqhkjOPQQDAgNHADBEAiBwh41Cz3rLZ4lg\nuzqhS7BEc14i4+rPGSObUFEbwEQcxwIgBQ3BPWFmuHI5HHM1W9uJFOlQEQ2/jZwT\nOQ85HzjGAIM=\n-----END CERTIFICATE-----\n",
"ssl_certfile": "-----BEGIN CERTIFICATE-----\nMIICwTCCAmegAwIBAgIQKD3Y+JXk8nqpsKLTjxW3ETAKBggqhkjOPQQDAjAiMSAw\nHgYDVQQDExdjbHVzdGVyLWludGVybWVkaWF0ZS1jYTAeFw0yMjAxMTkwODIxMzFa\nFw0yMjA0MTkwODIxMzFaMD4xPDA6BgNVBAMTM2lwbC1vcGVyYXRvci1jb250cm9s\nbGVyLW1hbmFnZXIuaXBsLW9wZXJhdG9yLXN5c3RlbTCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBANjYk1clGK1RFSJ009Rv0zX8ZArXnHabaWouznyippWY\nVu6P9m7R4c+a1baWCa/rTV/4+CgqKrZ5qlFpA1YVvAA47Dz40VJ+d8r9w8Sic9kG\nUr4rsBOA9yg00tMeAUothPHfaI/IIPd1gDMqTEqEWmqQ/lJpv1/COSxNmuUf3jC7\ny+luEN7yW4gUtyytxT4ZGkJcP47guHmJ0KIyzp8fJDXojgJVTK+A067q5/dJCaPv\nKze2GA0oZr/RMe6Q0PUMxQnNWZgfeQs3t4lu4pZJYwL2wqrkmvYhX2/bdY93H911\nNWgiuFozRoIIykFdKnG6HUHS1xygTmqq28zil0MkfP0CAwEAAaOBlzCBlDAOBgNV\nHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAf\nBgNVHSMEGDAWgBReM9YQsPK8Gy/ioQccN3XcTTtkEDA+BgNVHREENzA1gjNpcGwt\nb3BlcmF0b3ItY29udHJvbGxlci1tYW5hZ2VyLmlwbC1vcGVyYXRvci1zeXN0ZW0w\nCgYIKoZIzj0EAwIDSAAwRQIhAN6P7szDlR5ibJuEPJII/FhV3OJoIbVBzVoAy4zo\n1u5DAiAqRCi8K6px5W4PKQllWvwqBJwkssr1/Tg/9TFkbi9aCA==\n-----END CERTIFICATE-----\n",
"ssl_check_hostname": true,
"ssl_ciphers": null,
"ssl_crlfile": null,
"ssl_keyfile": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"ssl_password": null,
"ssl_supported_protocols": null
}

Specifications

Library version: 0.15.0
Result of pip list command:
Kafka version: 2.8.1
Python version: 3.8
OS: 5.15.12-100.fc34.x86_64

StephenSorriaux commented 2 years ago

Hello @david-freistrom,

Thank you for this issue, it seems like some topic does not have any min.insync.replicas defined. I will dig more.

david-freistrom commented 2 years ago

Maybe it's because I had not the right ACL permissions for that topic?!? The error message is kind of confusing.

ryarnyah commented 2 years ago

I digged more and if you do not have sufficient permission you cannot get topics configuration and that is the bug.

StephenSorriaux / ansible-kafka-admin