Key deployment code for Vagrant saltstack consistently fails

[torenware]

The key deployment code in master/Section5/Video3/toplevel/vagrant/Vagrantfile and similar for Section 1 / video 23 and 24 consistently fail for me when called as

vagrant up master && vagrant up minion

for new VMs. The issue is reported on the Udemy discussion section for the Packt course. I've posted the error as it appears, and also posted what you find in /var/log/salt for the master and for the minion.

While the key problem can be fixed manually, note that the ServerSpec part of the course won't run if this isn't fixed, since the spec script will founder on the problem, since ServerSpec needs to create a working saltstack install automatically.

[dmangot]

@torenware thanks for the report. I've actually taken a stab at fixing this twice already. I'm a little bit confused as to how SaltStack provisioning has changed. Possibly some new interaction between their bootstrap script and the provider. One possibility would be just to write my own bootstrap script, not sure if that would be better or not.

Hopefully I can take a swing at this again and it will be resolved the 3rd time.

[torenware]

@dmangot -- thanks for getting back to me. You might want to take a look at https://github.com/UtahDave/salt-vagrant-demo. His code is actually simpler, and I've verified it works right out of the box; I just cloned his repo's master, ran vagrant up, and 3 virtuals came up with the keys correctly installed on the 'master' instance. On the assumption that the licenses are compatible, his approach is likely the path of least resistance.

Thanks for the course. Any misgivings I have about it as it exists has more to do with Packt than you, to be clear.

[dmangot]

@torenware Finally got a chance to sit down and debug this. I'm pretty sure the problem was ordering when using multiple provisioners (https://www.vagrantup.com/docs/provisioning/basic_usage.html#multiple-provisioners). I've simplified the Vagrantfile and removed the shell provisioners and followed more of the pattern from the repo you linked. I'm not super excited about duplicating the minon config with a few bytes changed but I think this will be more stable.

Let me know if this solves your problem. If so, I can propagate the changes to the other sections where the same issue exists.

Thanks very much for opening this issue.

[torenware]

Dave,

Thanks so much for getting back to me on this.

I’ll give it a try.

A bit frustrated with the Packt folks, but it’s not a problem with what you’re doing. Very good course.

Thanks, Rob

On Aug 10, 2017, at 3:31 PM, Dave Mangot notifications@github.com wrote:

@torenware https://github.com/torenware Finally got a chance to sit down and debug this. I'm pretty sure the problem was ordering when using multiple provisioners (https://www.vagrantup.com/docs/provisioning/basic_usage.html#multiple-provisioners https://www.vagrantup.com/docs/provisioning/basic_usage.html#multiple-provisioners). I've simplified the Vagrantfile and removed the shell provisioners and followed more of the pattern from the repo you linked. I'm not super excited about duplicating the minon config with a few bytes changed but I think this will be more stable.

Let me know if this solves your problem. If so, I can propagate the changes to the other sections where the same issue exists.

Thanks very much for opening this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmangot/Mastering-DevOps/issues/3#issuecomment-321691236, or mute the thread https://github.com/notifications/unsubscribe-auth/AATqZZ9uguo8uLwiqVV1JEQolggK4q57ks5sW4TagaJpZM4OrzYY.

[torenware]

OK, we still have a similar, but not an identical fail. Doing a complete fresh run of vagrant up where there's no existing virtual (i.e., I ran vagrant destroy master last week),

The error portion of the vagrant up run looks like this:

Setting up python-git (0.3.2~RC1-3) ...
Setting up salt-master (2017.7.0+ds-1) ...

Configuration file '/etc/salt/master'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
 ==> Using current old file as you requested.
salt-master start/running, process 6115
Processing triggers for ureadahead (0.100.0-16) ...
 *  INFO: Running install_ubuntu_stable_post()
 Adding system startup for /etc/init.d/salt-master ...
   /etc/rc0.d/K20salt-master -> ../init.d/salt-master
   /etc/rc1.d/K20salt-master -> ../init.d/salt-master
   /etc/rc6.d/K20salt-master -> ../init.d/salt-master
   /etc/rc2.d/S20salt-master -> ../init.d/salt-master
   /etc/rc3.d/S20salt-master -> ../init.d/salt-master
   /etc/rc4.d/S20salt-master -> ../init.d/salt-master
   /etc/rc5.d/S20salt-master -> ../init.d/salt-master
 System start/stop links for /etc/init.d/salt-minion already exist.
 *  INFO: Running install_ubuntu_check_services()
 *  INFO: Running install_ubuntu_restart_daemons()
salt-master stop/waiting
salt-master start/running, process 6753
salt-minion start/running, process 6758
 *  INFO: Running daemons_running()
 * ERROR: salt-minion was not found running
 * ERROR: Failed to run daemons_running()!!!
 * ERROR: salt-master was not found running. Pass '-D' to bootstrap-salt.sh when bootstrapping for additional debugging information...
 * ERROR: salt-minion was not found running. Pass '-D' to bootstrap-salt.sh when bootstrapping for additional debugging information...

The log files on the master look very similar to before:

root@vagrant-ubuntu-trusty-64:/var/log/salt# ls -l
total 8
-rw-r--r-- 1 root root   0 Aug 10 22:50 key
-rw-r----- 1 root root 493 Aug 10 22:47 master
-rw-r----- 1 root root 871 Aug 10 22:47 minion
root@vagrant-ubuntu-trusty-64:/var/log/salt# cat master
2017-08-10 22:47:00,322 [salt.transport.mixins.auth][ERROR   ][6156] Authentication attempt from minion failed, the public keys did not match. This may be an attempt to compromise the Salt cluster.
2017-08-10 22:47:04,416 [salt.utils.parsers][WARNING ][6115] Master received a SIGTERM. Exiting.
2017-08-10 22:47:11,522 [salt.transport.mixins.auth][ERROR   ][6791] Authentication attempt from master failed, the public keys did not match. This may be an attempt to compromise the Salt cluster.
root@vagrant-ubuntu-trusty-64:/var/log/salt# cat minion
2017-08-10 22:45:56,659 [salt.utils.parsers][WARNING ][3282] Minion received a SIGTERM. Exiting.
2017-08-10 22:46:37,188 [salt.minion      ][ERROR   ][3337] Error while bringing up minion for multi-master. Is master at 192.168.50.10 responding?
2017-08-10 22:47:00,340 [salt.crypt       ][CRITICAL][3337] The Salt Master has rejected this minion's public key!
To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.
Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.
2017-08-10 22:47:11,524 [salt.crypt       ][CRITICAL][6765] The Salt Master has rejected this minion's public key!
To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.
Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.
root@vagrant-ubuntu-trusty-64:/var/log/salt#

I get this when I run 'salt-key -L':

root@vagrant-ubuntu-trusty-64:~# salt-key -L
Accepted Keys:
master
minion
Denied Keys:
master
minion
Unaccepted Keys:
Rejected Keys:

Before the fix, I got the following:

root@vagrant-ubuntu-trusty-64:~# salt-key -L
Accepted Keys:
master
minion
Denied Keys:
master
Unaccepted Keys:
vagrant-ubuntu-trusty-64
Rejected Keys:

Pulling the keys with "salt-key -P" is also similar, but not identical:

Accepted Keys:
master:  -----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA/R+/95lokbMzcYBH0WkD
qfx84p7YKVhDUrlhDXjG7ac3lK0zKKPJ8VNzhBZgF+Ir3E37RBTHeMQdPVTowDVd
ozZGHU9DXMiqUhbHyqPVyXQ5qM9Sf3OeIxTelD7o+p39PgAT0bxcByt08eeGtkKY
ENY4RmEKsVgF7gsgcobXxROnasumzkXSWd6Sz01MeUSaajpINSn8TgwKkbzgz+VL
fjBomgRRYiLG4eDEERWy1Hw30DpB7izMfjFW2wDu6ctAjGJFlaqy8eIriVfz1P0S
uDX0id8pso8KVjODT/qjPPopFY7WtvRIbcDbuQBdruHxXLPJj1cveeSKDewu8Mil
cQIDAQAB
-----END PUBLIC KEY-----

minion:  -----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxjxuB4VchKOlC7o4OaoY
C07Rb532xe80//u1xOPfDUrSYCtnHImd24TpliQFbWE8sXDiWG35ZRzXff9IS7+C
cHj/Ih+0OHpLSpHx5NOiDWDHIClIoVMrun1QoUPa0wwEeIvOnouB11d8emBPreZl
D4NeAuJSKlk4HKl2mALtq8tR+wWldGPsb6ctTDa5cGK9glj9WIk3TvM6aGLn+Lhh
0x3vRnCda7m62yDgyQsK8GB6hr4gX8TbfSjKe+du7eZPDcprFNqwC6lWajaSDX/N
jJTNm28Rha4G/gLtXeBdbtQesSarYQaTqy5FRqsvxrYpY3LUCycbh00sGTgwSxnG
eQIDAQAB
-----END PUBLIC KEY-----

Denied Keys:
master:  -----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA705v7Srv1h/yxzLchToo
Xtq/ywOnSRQ9aZSHTL6jTloDQqFCunj3FxtHGxqxZS62DJ3XIrW/2DQeRkoef5SS
k7AzYjiTgnR+fT+XsUun4JTjwSaD0A71wbRKUOZOXgiu3QMkUY2gdOZk8+1bpllG
eUBoRhwo5qhZFMMvncKKzlz0NmcC1YupfBcIIJedoiSYR4z/vzRGaK+DpBsz2dgV
vFcIeHGo6Bp1hvzdXeDI25OFFzuj6l7ZwagW4VIhX7HSQz9wc0KZH0qYrj3C5N3n
c1E15bWVBeW8OA8lIySIaFRK+ZASsa9z+tYJ27hevAJcsM4vwgFGY2ePNbWnwSTR
JwIDAQAB
-----END PUBLIC KEY-----
minion:  -----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA705v7Srv1h/yxzLchToo
Xtq/ywOnSRQ9aZSHTL6jTloDQqFCunj3FxtHGxqxZS62DJ3XIrW/2DQeRkoef5SS
k7AzYjiTgnR+fT+XsUun4JTjwSaD0A71wbRKUOZOXgiu3QMkUY2gdOZk8+1bpllG
eUBoRhwo5qhZFMMvncKKzlz0NmcC1YupfBcIIJedoiSYR4z/vzRGaK+DpBsz2dgV
vFcIeHGo6Bp1hvzdXeDI25OFFzuj6l7ZwagW4VIhX7HSQz9wc0KZH0qYrj3C5N3n
c1E15bWVBeW8OA8lIySIaFRK+ZASsa9z+tYJ27hevAJcsM4vwgFGY2ePNbWnwSTR
JwIDAQAB
-----END PUBLIC KEY-----

So it looks like a generated key is still getting in front of you somehow. I'm going to leave the master running, if you can think of anything else you want me to look at.

[torenware]

I'll take a look at the new Vagrantfile as well. IIRC I got his script to load the keys, although I didn't add your tutorial related code for Video3. I'll start from his file, and add your stuff in incrementally, to see if I can get it working.

Again, if you need to see anything on the partially built virtual, ask and I'll see if I can get it for you.

[dmangot]

This might be a long shot but can you try a vagrant box remove ubuntu/trusty64 before running the vagrant up? I was getting some weird behavior with the keys until I had it download a fresh Ubuntu.

[torenware]

Sure. I’ll give it a try tomorrow.

R

On Aug 10, 2017, at 6:01 PM, Dave Mangot notifications@github.com wrote:

This might be a long shot but can you try a vagrant box remove ubuntu/trusty64 before running the vagrant up? I was getting some weird behavior with the keys until I had it download a fresh Ubuntu.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmangot/Mastering-DevOps/issues/3#issuecomment-321711405, or mute the thread https://github.com/notifications/unsubscribe-auth/AATqZYUDhAGY3tQCwxMVjTjDiWcSZoX9ks5sW6fRgaJpZM4OrzYY.

[torenware]

OK, took a couple of extra days. vagrant destroy master; vagrant box remove ubuntu/trusty64; and then, vagrant up. Same fail as before. Keys are as the previous attempt.

So, no joy yet.

Thanks, Rob

On Aug 10, 2017, at 11:19 PM, Rob Thorne rob@torenware.com wrote:

Sure. I’ll give it a try tomorrow.

R

On Aug 10, 2017, at 6:01 PM, Dave Mangot <notifications@github.com mailto:notifications@github.com> wrote:

This might be a long shot but can you try a vagrant box remove ubuntu/trusty64 before running the vagrant up? I was getting some weird behavior with the keys until I had it download a fresh Ubuntu.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmangot/Mastering-DevOps/issues/3#issuecomment-321711405, or mute the thread https://github.com/notifications/unsubscribe-auth/AATqZYUDhAGY3tQCwxMVjTjDiWcSZoX9ks5sW6fRgaJpZM4OrzYY.

[dmangot]

What version of Vagrant? What OS? (yeah, just trying to see if I can reproduce)

[torenware]

On Aug 14, 2017, at 6:09 PM, Dave Mangot notifications@github.com wrote:

What version of Vagrant? What OS? (yeah, just trying to see if I can reproduce)

$ vagrant version Installed Version: 1.9.3 Latest Version: 1.9.7

MacOS 10.11.6 (El Capitan)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmangot/Mastering-DevOps/issues/3#issuecomment-322350548, or mute the thread https://github.com/notifications/unsubscribe-auth/AATqZRfKdqJstyNfAGuGBkrE4x-QAMLQks5sYO-0gaJpZM4OrzYY.

[torenware]

Also, not sure if this is relevant, but it looks like Vagrant is using the Mac’s preinstalled copy of ruby:

$ ruby -v ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin15] Robs-MacBook-Pro-3:diaspora rtoren$ type ruby ruby is hashed (/usr/bin/ruby)

On Aug 14, 2017, at 6:11 PM, Rob Thorne rob@torenware.com wrote:

On Aug 14, 2017, at 6:09 PM, Dave Mangot <notifications@github.com mailto:notifications@github.com> wrote:

What version of Vagrant? What OS? (yeah, just trying to see if I can reproduce)

$ vagrant version Installed Version: 1.9.3 Latest Version: 1.9.7

MacOS 10.11.6 (El Capitan)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmangot/Mastering-DevOps/issues/3#issuecomment-322350548, or mute the thread https://github.com/notifications/unsubscribe-auth/AATqZRfKdqJstyNfAGuGBkrE4x-QAMLQks5sYO-0gaJpZM4OrzYY.

[dmangot]

Right now I'm on Vagrant 1.8.5, and Vagrant is supposed to use a bundled Ruby, not the system Ruby I think (http://mitchellh.com/abandoning-rubygems). I will try and upgrade to the latest Vagrant. My Ruby comes up the same as yours.

[dmangot]

I did a bunch of testing. There definitely seems to be a problem once you get to Vagrant 1.8.7. It's complaining about strings vs. integers in the salt provisioner itself (i.e. not the Vagrantfile AFAICT). I've submitted a patch that pins the version at 1.8.5 or 1.8.6, both of which I've tested to work successfully.

[frostbyte2013]

Thanks Dave! This was holding me up. I can confirm, working with 1.8.6 - ran into one issue after previously having 1.9.7 installed, you may receive the following nag:

Bundler, the underlying system used to manage Vagrant plugins, is reporting that a plugin or its dependency can't be found. This is usually caused by manual tampering with the 'plugins.json' file in the Vagrant home directory. To fix this error, please remove that file and reinstall all your plugins using vagrant plugin install.

I had to remove the following vagrant file to resolve: rm -rf ~/.vagrant.d

[dmangot]

@torenware may I close this issue?

-Dave

[torenware]

@dmangot -- if you're willing to leave it open a bit longer, much obliged. I actually never got this working with the current code. I've been doing other things, but I think that factoring the alternative version so that it runs the equivalent actions is likely the right fix.