Update to Latest Ubuntu Server

[tomjirsa]

Update deploy to latest Ubuntu Server 16.04.1 LTS https://www.ubuntu.com/download/server

[tomjirsa]

FIrst error in deployment:

TASK [kafka : Start Kafka server] **********************************************
fatal: [producer]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service kafka-broker: cannot enable"}

[jpaulovic]

Ubuntu 16.x uses systemd instead of upstart. You need to change the init script for Kafka to systemd format, example: https://www.digitalocean.com/community/questions/convert-run-at-startup-script-from-upstart-to-systemd-for-ubuntu-16

[tomjirsa]

Systemd for kafka start:

[Unit] Description=Apache Kafka server (broker) Documentation=http://kafka.apache.org/documentation.html Requires=network.target remote-fs.target After=network.target remote-fs.target`

[Service] Type=simple PIDFile=/var/run/kafka.pid User=kafka Group=kafka ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties ExecStop=/opt/kafka/bin/kafka-server-stop.sh Restart=on-failure SyslogIdentifier=kafka

[Install] WantedBy=multi-user.target

Systemd script for zookeeper:

[Unit] Description=Apache Zookeeper server (broker) Documentation=https://zookeeper.apache.org/ Requires=network.target remote-fs.target After=network.target remote-fs.target

[Service] Type=simple PIDFile=/var/run/zookeeper.pid User=kafka Group=kafka ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh Restart=on-failure SyslogIdentifier=zookeeper

[Install] WantedBy=multi-user.target

[tomjirsa]

Kafka needs to run as root - needs to check [Unit] Description=Apache Kafka server (broker) Documentation=http://kafka.apache.org/documentation.html Requires=network.target remote-fs.target After=network.target remote-fs.target`

[Service] Type=simple PIDFile=/var/run/kafka.pid User=root Group=root ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties ExecStop=/opt/kafka/bin/kafka-server-stop.sh Restart=on-failure SyslogIdentifier=kafka

[Install] WantedBy=multi-user.target

[tomjirsa]

zookeeper does not need systemd script. Kafka issue solved

[tomjirsa]

IPfixcol fail:

TASK [ipfixcol : Copy upstart files for IPFIXcol (1/2)] ************************ fatal: [security-cloud-producer]: FAILED! => {"changed": true, "failed": true, "gid": 0, "group": "root", "mode": "0644", "msg": "chown failed: failed to look up user vagrant", "owner": "root", "path": "/etc/default/ipfixcol", "size": 58, "state": "file", "uid": 0}

[tomjirsa]

Systemd script for ipfixcol (need to check priviledges) [Unit] Description=Ipfixcol service Documentation=https://github.com/CESNET/ipfixcol Requires=network.target remote-fs.target After=network.target remote-fs.target

[Service] Type=simple PIDFile=/var/run/ipfixcol.pid User=root Group=root EnvironmentFile=/etc/default/ipfixcol ExecStart=/usr/local/bin/ipfixcol -c $IPFIXCOL_SCRIPT -v 2 ExecStop= Restart=on-failure SyslogIdentifier=ipfixcol

[Install] WantedBy=multi-user.target

Fixed

[tomjirsa]

TASK [example-application : Install ujson dependencies] **** failed: [security-cloud-master] (item=[u'cpp-4.6', u'gcc-4.6-base', u'python-dev']) => {"failed": true, "item": ["cpp-4.6", "gcc-4.6-base", "python-dev"], "msg": "No package matching 'cpp-4.6' is available"}

[tomjirsa]

can be fixed using http://stackoverflow.com/questions/36283915/cannot-install-pip-packages-due-to-locale-error-inside-ubuntu-vagrant-box

export LC_CTYPE="en_US.UTF-8" locale-gen en_US.UTF-8 sudo dpkg-reconfigure locales

[tomjirsa]

vagrantbox ubuntu/xenial64 does not contain python2

• temporary solution added python2 instalation into vagrant file

 # First, install python
  config.vm.provision "shell" do |s|
    s.inline = "apt-get install -y python"
  end

• stable solution: #7

[xdanos]

Not a good idea, kafka should run as kafka/kafka

User=root
Group=root

[tomjirsa]

Not a good idea, kafka should run as kafka/kafka Agree, changed, fixed in d28a1a5d52b6a418ec8147b78d095357acc0dbec @tomjirsa tomjirsa committed 4 minutes ago

[xdanos]

TCP Normalizer included in 3e9ed7ffe7f363650f271629bea519ea04fd86eb for issue #8

[tomjirsa]

Merged to master, closing