CSIRT-MU / Stream4Flow

A framework for the real-time network traffic analysis based on world-leading technologies for distributed stream processing, network traffic monitoring, and visualization.
https://csirt.muni.cz/?lang=en
MIT License
99 stars 36 forks source link

Kafka shows warning on output #100

Closed linuxxxer closed 4 years ago

linuxxxer commented 4 years ago

Hello I was running standalone deployment, when multiply issues came up. The first was, that Java version 12 is not available, so I needed to change the configuration file to download Java 13 instead of 12 - this looks like solved it. The next were with Kafka. While provisioning this error message showed up:

_"TASK [kafka : Add Kafka topic "ipfix.entry"] *** fatal: [producer]: FAILED! => {"msg": "The conditional check 'resultin|failed' failed. The error was: template error while templating string: no filter named 'failed'. String: {% if resultin|failed %} True {% else %} False {% endif %}\n\nThe error appears to be in '/vagrant/ansible/roles/kafka/tasks/start.yml': line 22, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Add Kafka topic \"ipfix.entry\"\n ^ here\n"}

RUNNING HANDLER [ubuntu-systemd-normalizer : Restart Service Instance 'default'] ***

PLAY RECAP ***** producer : ok=27 changed=21 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

Ansible failed to complete successfully. Any error output should be visible above. Please fix these errors and try again."_

When I searched the configuration file and tried to run the command manually through ssh, it seemed no errors occured, but there was a warning message:

"vagrant@producer:~$ /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic ipfix.entry --partitions 1 --replication-factor 1 WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('') could collide. To avoid issues it is best to use either, but not both. Created topic ipfix.entry."_

I tried to run "vagrant provision" once more and I got the same error. After commenting out the section, the next section was doing similar trouble (- name: Set retention on the input topic). Again I tried to run the command manually after I searched for the value "retention". "vagrant@producer:~$ /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic ipfix.entry --config retention.ms=30000 WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases. Going forward, please use kafka-configs.sh for this functionality Updated config for topic ipfix.entry."

Can you help me to check about that? Is it only because of new Kafka version, or I am doing something wrong?

cermmik commented 4 years ago

The error was caused by an older version of the provisioning template incompatible with the new version of Ansible.