`vagrant up`, `ansible` warnings and fatal errors

[rickhg12hs]

I want to try NEMEA and so I followed the recommendation at https://nemea.liberouter.org/doc/#usage-by-user by using the vagrant/ansible option.

Here's what happened on a Fedora 29 host after following the instructions.

$ vagrant validate
==> vagrant: A new version of Vagrant is available: 2.2.6!
==> vagrant: To upgrade visit: https://www.vagrantup.com/downloads.html

Vagrantfile validated successfully.
$ vagrant version
Installed Version: 2.1.2
Latest Version: 2.2.6

To upgrade to the latest version, visit the downloads page and
download and install the latest version of Vagrant from the URL
below:

  https://www.vagrantup.com/downloads.html

If you're curious what changed in the latest release, view the
CHANGELOG below:

  https://github.com/hashicorp/vagrant/blob/v2.2.6/CHANGELOG.md
$ vagrant status
Current machine states:

list-vagrant              not created (libvirt)

The Libvirt domain is not created. Run `vagrant up` to create it.
$ vagrant up
Bringing machine 'list-vagrant' up with 'libvirt' provider...
==> list-vagrant: Box 'centos/7' could not be found. Attempting to find and install...
    list-vagrant: Box Provider: libvirt
    list-vagrant: Box Version: >= 0
==> list-vagrant: Loading metadata for box 'centos/7'
    list-vagrant: URL: https://vagrantcloud.com/centos/7
==> list-vagrant: Adding box 'centos/7' (v1905.1) for provider: libvirt
    list-vagrant: Downloading: https://vagrantcloud.com/centos/boxes/7/versions/1905.1/providers/libvirt.box
    list-vagrant: Download redirected to host: cloud.centos.org
==> list-vagrant: Successfully added box 'centos/7' (v1905.1) for 'libvirt'!
==> list-vagrant: Uploading base box image as volume into libvirt storage...
==> list-vagrant: Creating image (snapshot of base box volume).
==> list-vagrant: Creating domain with the following settings...
==> list-vagrant:  -- Name:              vagrant_list-vagrant
==> list-vagrant:  -- Domain type:       kvm
==> list-vagrant:  -- Cpus:              1
==> list-vagrant:  -- Feature:           acpi
==> list-vagrant:  -- Feature:           apic
==> list-vagrant:  -- Feature:           pae
==> list-vagrant:  -- Memory:            512M
==> list-vagrant:  -- Management MAC:    
==> list-vagrant:  -- Loader:            
==> list-vagrant:  -- Base box:          centos/7
==> list-vagrant:  -- Storage pool:      default
==> list-vagrant:  -- Image:             /var/lib/libvirt/images/vagrant_list-vagrant.img (41G)
==> list-vagrant:  -- Volume Cache:      default
==> list-vagrant:  -- Kernel:            
==> list-vagrant:  -- Initrd:            
==> list-vagrant:  -- Graphics Type:     vnc
==> list-vagrant:  -- Graphics Port:     5900
==> list-vagrant:  -- Graphics IP:       127.0.0.1
==> list-vagrant:  -- Graphics Password: Not defined
==> list-vagrant:  -- Video Type:        cirrus
==> list-vagrant:  -- Video VRAM:        9216
==> list-vagrant:  -- Sound Type:   
==> list-vagrant:  -- Keymap:            en-us
==> list-vagrant:  -- TPM Path:          
==> list-vagrant:  -- INPUT:             type=mouse, bus=ps2
==> list-vagrant: Creating shared folders metadata...
==> list-vagrant: Starting domain.
==> list-vagrant: Waiting for domain to get an IP address...
==> list-vagrant: Waiting for SSH to become available...
    list-vagrant: 
    list-vagrant: Vagrant insecure key detected. Vagrant will automatically replace
    list-vagrant: this with a newly generated keypair for better security.
    list-vagrant: 
    list-vagrant: Inserting generated public key within guest...
    list-vagrant: Removing insecure key from the guest if it's present...
    list-vagrant: Key inserted! Disconnecting and reconnecting using new SSH key...
==> list-vagrant: Forwarding ports...
==> list-vagrant: 443 (guest) => 8443 (host) (adapter eth0)
==> list-vagrant: 22 (guest) => 22222 (host) (adapter eth0)
==> list-vagrant: 5555 (guest) => 5555 (host) (adapter eth0)
==> list-vagrant: 4739 (guest) => 4739 (host) (adapter eth0)
==> list-vagrant: Configuring and enabling network interfaces...
    list-vagrant: SSH address: 192.168.121.203:22
    list-vagrant: SSH username: vagrant
    list-vagrant: SSH auth method: private key
==> list-vagrant: Rsyncing folder: /home/rick/ČVUT/B191/GitHub-CESNET-List/LiST/vagrant/ => /vagrant
==> list-vagrant: Running provisioner: ansible...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.9.1).

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode

    list-vagrant: Running ansible-playbook...
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to 
allow bad characters in group names by default, this will change, but still be 
user configurable on deprecation. This feature will be removed in version 2.10.
 Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

[DEPRECATION WARNING]: 'include' for playbook includes. You should use 
'import_playbook' instead. This feature will be removed in version 2.12. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.

PLAY [list] ********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [list-vagrant]

TASK [settings : Set default hostname from inventory file] *********************
skipping: [list-vagrant]

TASK [settings : Set system hostname] ******************************************
changed: [list-vagrant]

TASK [settings : Set hostname to /etc/hosts] ***********************************
changed: [list-vagrant]

TASK [settings : Set SELinux policy to permissive] *****************************
changed: [list-vagrant]

TASK [settings : Set large system receive buffers using sysctl] ****************
changed: [list-vagrant]

TASK [settings : Activate large system receive buffers] ************************
changed: [list-vagrant]

TASK [settings : Set timezone] *************************************************
changed: [list-vagrant]

TASK [common : Check for compatible system] ************************************
ok: [list-vagrant]

TASK [common : Install necessary packages for Ansible run] *********************
changed: [list-vagrant]

TASK [common : Expand the ansible ssh private key file path] *******************
ok: [list-vagrant]

TASK [epel : Add Epel repository] **********************************************
changed: [list-vagrant]

TASK [apache : Install Apache] *************************************************
changed: [list-vagrant]

TASK [apache : Redirect HTTP to HTTPs] *****************************************
changed: [list-vagrant]

TASK [apache : Check for certificates dir] *************************************
ok: [list-vagrant -> localhost]

TASK [apache : Copy certificates] **********************************************
skipping: [list-vagrant]

TASK [apache : Create fallback certificate dir] ********************************
changed: [list-vagrant]

TASK [apache : Fallback to local certificate] **********************************
changed: [list-vagrant]

TASK [apache : Fallback to local certificate key] ******************************
changed: [list-vagrant]

TASK [apache : Configure Apache SSL] *******************************************
changed: [list-vagrant]

TASK [apache : Start & enable Apache] ******************************************
changed: [list-vagrant]

TASK [mongodb : Add Mongo repository] ******************************************
changed: [list-vagrant]

TASK [mongodb : Install Mongo, python, etc] ************************************
changed: [list-vagrant]

TASK [mongodb : Copy mongod configuration] *************************************
changed: [list-vagrant]

TASK [mongodb : Start & enable Mongo] ******************************************
changed: [list-vagrant]

TASK [pip3.4 : Install dependencies] *******************************************
changed: [list-vagrant]

TASK [pip3.4 : Get python pip] *************************************************
changed: [list-vagrant]

TASK [pip3.4 : Install python pip] *********************************************
changed: [list-vagrant]

TASK [warden-client : Install warden client] ***********************************
changed: [list-vagrant]

TASK [warden-client : Copy Warden filer python file] ***************************
changed: [list-vagrant]

TASK [ipfixcol : Add IPFIXcol repo] ********************************************
changed: [list-vagrant]

TASK [ipfixcol : Add NEMEA repo (unirec dependency)] ***************************
changed: [list-vagrant]

TASK [ipfixcol : Install IPFIXcol] *********************************************
changed: [list-vagrant]

TASK [ipfixcol : Check for host specific startup.xml] **************************
ok: [list-vagrant -> localhost]

TASK [ipfixcol : Select host specific startup.xml] *****************************
ok: [list-vagrant]

TASK [ipfixcol : Copy startup.xml configuration template] **********************
changed: [list-vagrant]

TASK [nemea : Add NEMEA repository] ********************************************
ok: [list-vagrant]

TASK [nemea : Add NEMEA-Testing repository] ************************************
skipping: [list-vagrant]

TASK [nemea : Install NEMEA packages] ******************************************
changed: [list-vagrant]

TASK [nemea : Get sample data] *************************************************
[WARNING]: Consider using the file module with state=directory rather than
running 'mkdir'.  If you need to use command because file is insufficient you
can add 'warn: false' to this command task or set 'command_warnings=False' in
ansible.cfg to get rid of this message.

changed: [list-vagrant]

TASK [nemea : Copy host specific NEMEA configuration] **************************
fatal: [list-vagrant]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --delete-after --recursive --times --rsh=/usr/bin/ssh -S none -i '/home/rick/ČVUT/B191/GitHub-CESNET-List/LiST/vagrant/.vagrant/machines/list-vagrant/virtualbox/private_key' -o Port=22222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --rsync-path=sudo rsync --omit-dir-times --exclude=supervisor_config_gener.xml --exclude=nemea_status.conf --out-format=<<CHANGED>>%i %n%L /home/rick/ČVUT/B191/GitHub-CESNET-List/LiST/ansible/inventory/host_files/list-vagrant/nemea/ vagrant@localhost:/etc/nemea/", "msg": "Warning: Identity file /home/rick/ČVUT/B191/GitHub-CESNET-List/LiST/vagrant/.vagrant/machines/list-vagrant/virtualbox/private_key not accessible: No such file or directory.\nWarning: Permanently added '[localhost]:22222' (ECDSA) to the list of known hosts.\r\nvagrant@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).\r\nrsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]\n", "rc": 255}
...ignoring

TASK [nemea : Set NEMEA permissions] *******************************************
changed: [list-vagrant]

TASK [nemea : Check for reporters configuration] *******************************
ok: [list-vagrant]

TASK [nemea : Create default reporters configuration] **************************
changed: [list-vagrant]

TASK [nemea : Copy logrotate configuration configuration] **********************
changed: [list-vagrant]

TASK [nemea : Check for Warden config dir] *************************************
ok: [list-vagrant -> localhost]

TASK [nemea : Copy Warden client configuration for NEMEA] **********************
skipping: [list-vagrant]

TASK [nemea : Set Warden config dir ownership] *********************************
skipping: [list-vagrant]

TASK [nemea : Set Warden config dir permissions] *******************************
skipping: [list-vagrant]

TASK [nemea : Add configuration to startup.xml] ********************************
changed: [list-vagrant]

TASK [nemea : Linkload monitor] ************************************************
changed: [list-vagrant]

TASK [nemea : Start NEMEA supervisor] ******************************************
changed: [list-vagrant]

TASK [nemea-dashboard : Install dependencies] **********************************
ok: [list-vagrant]

TASK [nemea-dashboard : Clone Nemea-Dashboard] *********************************
[WARNING]: Your git version is too old to fully support the depth argument.
Falling back to full checkouts.

changed: [list-vagrant]

TASK [nemea-dashboard : Install requirements for dashboard] ********************
fatal: [list-vagrant]: FAILED! => {"changed": false, "cmd": ["/bin/pip3", "install", "-r", "/var/www/html/Nemea-Dashboard/requirements.txt"], "msg": "stdout: Collecting Flask==1.0.3 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 1))\n  Downloading https://files.pythonhosted.org/packages/9a/74/670ae9737d14114753b8c8fdf2e8bd212a05d3b361ab15b44937dfd40985/Flask-1.0.3-py2.py3-none-any.whl (92kB)\nCollecting Flask-Cors==3.0.7 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 2))\n  Downloading https://files.pythonhosted.org/packages/65/cb/683f71ff8daa3aea0a5cbb276074de39f9ab66d3fbb8ad5efb5bb83e90d2/Flask_Cors-3.0.7-py2.py3-none-any.whl\nCollecting itsdangerous==1.1.0 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 3))\n  Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl\nCollecting Jinja2==2.10.1 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 4))\n  Downloading https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl (124kB)\nCollecting MarkupSafe==1.1.1 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 5))\n  Downloading https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl\nCollecting py-bcrypt==0.4 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 6))\n  Downloading https://files.pythonhosted.org/packages/68/b1/1c3068c5c4d2e35c48b38dcc865301ebfdf45f54507086ac65ced1fd3b3d/py-bcrypt-0.4.tar.gz\nCollecting pycparser==2.19 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 7))\n  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)\nCollecting PyJWT==1.7.1 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 8))\n  Downloading https://files.pythonhosted.org/packages/87/8b/6a9f14b5f781697e51259d81657e6048fd31a113229cf346880bb7545565/PyJWT-1.7.1-py2.py3-none-any.whl\nCollecting pymongo==3.8.0 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 9))\n  Downloading https://files.pythonhosted.org/packages/fb/4a/586826433281ca285f0201235fccf63cc29a30fa78bcd72b6a34e365972d/pymongo-3.8.0-cp36-cp36m-manylinux1_x86_64.whl (416kB)\nCollecting six==1.12.0 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 10))\n  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\nCollecting Werkzeug==0.15.4 (from -r /var/www/html/Nemea-Dashboard/requirements.txt (line 11))\n  Downloading https://files.pythonhosted.org/packages/9f/57/92a497e38161ce40606c27a86759c6b92dd34fcdb33f64171ec559257c02/Werkzeug-0.15.4-py2.py3-none-any.whl (327kB)\nCollecting click>=5.1 (from Flask==1.0.3->-r /var/www/html/Nemea-Dashboard/requirements.txt (line 1))\n  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)\nInstalling collected packages: Werkzeug, click, itsdangerous, MarkupSafe, Jinja2, Flask, six, Flask-Cors, py-bcrypt, pycparser, PyJWT, pymongo\n  Running setup.py install for py-bcrypt: started\n    Running setup.py install for py-bcrypt: finished with status 'error'\n    Complete output from command /usr/bin/python3 -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-bvh0lm1s/py-bcrypt/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))\" install --record /tmp/pip-7zivfqar-record/install-record.txt --single-version-externally-managed --compile:\n    running install\n    running build\n    running build_py\n    creating build\n    creating build/lib.linux-x86_64-3.6\n    creating build/lib.linux-x86_64-3.6/bcrypt\n    copying bcrypt/__init__.py -> build/lib.linux-x86_64-3.6/bcrypt\n    running build_ext\n    building 'bcrypt._bcrypt' extension\n    creating build/temp.linux-x86_64-3.6\n    creating build/temp.linux-x86_64-3.6/bcrypt\n    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c bcrypt/bcrypt.c -o build/temp.linux-x86_64-3.6/bcrypt/bcrypt.o\n    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c bcrypt/bcrypt_pbkdf.c -o build/temp.linux-x86_64-3.6/bcrypt/bcrypt_pbkdf.o\n    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c bcrypt/bcrypt_python.c -o build/temp.linux-x86_64-3.6/bcrypt/bcrypt_python.o\n    bcrypt/bcrypt_python.c:18:20: fatal error: Python.h: No such file or directory\n     #include \"Python.h\"\n                        ^\n    compilation terminated.\n    error: command 'gcc' failed with exit status 1\n    \n    ----------------------------------------\n\n:stderr: WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\nCommand \"/usr/bin/python3 -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-bvh0lm1s/py-bcrypt/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))\" install --record /tmp/pip-7zivfqar-record/install-record.txt --single-version-externally-managed --compile\" failed with error code 1 in /tmp/pip-build-bvh0lm1s/py-bcrypt/\n"}

RUNNING HANDLER [apache : Apache restart] **************************************
changed: [list-vagrant]

RUNNING HANDLER [nemea : Restart NEMEA supervisor] *****************************
changed: [list-vagrant]

PLAY RECAP *********************************************************************
list-vagrant               : ok=50   changed=39   unreachable=0    failed=1    skipped=6    rescued=0    ignored=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

[thorgrin]

From what I see there are two separate problems: 1) pip failed on compilation of bcrypt_python. Do you have libpython-devel package installed? I guess there might be a missing dependency. 2) The second problem is that the SSH private key for the vagrant VM is inaccessible:

"Warning: Identity file /home/rick/ČVUT/B191/GitHub-CESNET-List/LiST/vagrant/.vagrant/machines/list-vagrant/virtualbox/private_key not accessible: No such file or directory.

Could you look around the .vagrant directory to see where the private key is stored? It is possible that it's location has changed in some recent version. In that case, you'd just need to update ansible/inventory/hosts file with the new location.

[rickhg12hs]

I started again from the beginning after installing the latest vagrant rpm from hashicorp. It seems that my VirtualBox install was too new for my vagrant install. This created some issues because vagrant seemed to fallback to libvirt as the provider since it couldn't use VirtualBox. I don't really understand all the issues.

There was still a vagrant up failure because a build step failed when it couldn't find Python.h. After vagrant ssh, I sudo yum upgrade and then sudo yum install python-devel python3-devel python34-devel. Probably all *-devel weren't necessary, but I just want it to work.

After exit on the VM, I vagrant reload --provision on the host and it seemed to complete without error.

Is there a NEMEA sanity/smoke test that can be run to check if the VM is fully functional?

[thorgrin]

Thanks for testing this. I've just pushed a commit to fix the python34-devel issue. The problem arose when python newer than 3.4 became the default for pip3 in the system.

To see whether the system works, just open https://localhost:8443/scgui/ in your browser and you should see some data. To check the nemea output, use https://localhost:8443/liberouter-gui/nemea/events (admin/admin is the login). Moreover, the list of running modules should be available at https://localhost:8443/nemea-status/

[rickhg12hs]

Cool! It seems to work now! Thanks!

Where/what is the source for ipfixcol? And more generally, how do I explore/modify the NEMEA configuration? If I want to try to make a python module for NEMEA, is everything I need in this VM?

[thorgrin]

IPFIXcol is here: https://github.com/cesnet/ipfixcol It will be replaced by the new ipfixcol2 in near future, but ipfixcol2 is not included in LiST since the new one does not have all necessary features yet. However, if you only focus on Nemea framework, you can use the newer one, if you want to (the old one should be fine though).

For nemea, the documentation is right in the repo and on github pages:

• https://github.com/CESNET/Nemea (beware that there are some other repos as git submodules)
• https://nemea.liberouter.org/

As for whether everything is in the VM, well, no. You still have to checkout the nemea repositories and maybe install development tools (*-devel packages and maybe tools for compilation).