Linode: Alert_cpu_threshold error during provisioning.

[annwombat]

I am trying to deploy streisand on linode from an ubuntu vps server. I have seen it documented that local installs are not yet supported, but shouldn't it work if I'm coming from a completely different machine, even though it's not a workstation? But I am getting the localhost error:

=================================== Streisand will now set up your server. This process usually takes around ten minutes. Press Enter to begin setup... :

PLAY [Provision the Linode Server] ***

TASK [Gathering Facts] *** ok: [localhost]

TASK [genesis-linode : Get the default SSH key] ** changed: [localhost]

TASK [genesis-linode : Create the server] **** fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Alert_cpu_threshold must be between 0 and 2000."} to retry, use: --limit @/streisand/playbooks/linode.retry

PLAY RECAP *** localhost : ok=2 changed=1 unreachable=0 failed=1

[cpu]

@depluser What version of Ansible is installed on the Ubuntu VPS you are running Streisand from? What version of linode-python is installed?

[annwombat]

root@depl:~# ansible --version ansible 2.3.0.0 config file = configured module search path = Default w/o overrides python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]

How can I find out what version of linode-python is installed?

Thanks!

[cpu]

How can I find out what version of linode-python is installed?

You should be able to use dpkg -l linode-python.

[cpu]

Oops - I gave you bad advice. It's a pip package not a deb. Try:

pip show linode-python

[annwombat]

root@depl:~# pip show linode-python Name: linode-python Version: 1.1.1 Summary: Python bindings for Linode API Home-page: https://github.com/tjfontaine/linode-python Author: TJ Fontaine Author-email: tjfontaine@gmail.com License: UNKNOWN Location: /usr/local/lib/python2.7/dist-packages Requires:

[annwombat]

I was getting a warning about a pip upgrade being available, so I just went ahead and did the upgrade. Didn't change things tho - same error.

[cpu]

root@depl:~# ansible --version ansible 2.3.0.0

How did you install Ansible on the Ubuntu VM you're running Streisand from out of curiosity? 2.3.0.0 is newer than what I see packaged on the 16.04 server I checked.

I also see that Ansible 2.3's Linode module added new alert_cpu_threshold & alert_cpu_enabled arguments that may be related based on the error message you're seeing.

[annwombat]

Here is the command I used when installing ansible:

sudo pip install ansible markupsafe

The version of Ubuntu on the VM is 16.04 LTS

I had seen that page about the linode module - looks like the alert_cpu_threshold is optional? But how to turn it off??

[cpu]

sudo pip install ansible markupsafe

Can you try removing this & using the package from apt-get install ansible? It will help rule out this theory.

I had seen that page about the linode module - looks like the alert_cpu_threshold is optional? But how to turn it off??

Streisand doesn't provide this new argument to the Ansible Linode module so I'm not sure. It may be a red herring, we should test a different ansible version before going deeper on this theory.

[annwombat]

I was able to uninstall ansible with this command: sudo pip uninstall ansible

Then I installed it with: sudo apt-get install ansible

It installed 2.0.0.2 but now I have a new problem: root@depl:/streisand# ./streisand

S T R E I S A N D

Ansible 2.2.1.0 or higher is required.

[PaulColdren]

Haven't looked into the root cause, but in the interim you can get by with an older version of ansible and an older revision of this repo, just prior to #661 where the required ansible version was bumped to 2.3.0.0

I used:

pip install 'ansible<2.3.0.0'

to get an older ansible, and:

git reset --hard fa120fa6b172f13b0535434caa93d838d739762c

to get an older commit of the streisand repo (fa120fa6b172f13b0535434caa93d838d739762c) from before #661

[zweishar]

I also ran into this bug when deploying from MacOS Sierra 10.12.5

Taking the steps provided above worked as advertised, thanks @pcoldren :

• pip install 'ansible<2.3.0.0'
• git reset --hard fa120fa6b172f13b0535434caa93d838d739762c

[GlitchStorm]

@pcoldren Doesn't work for me, when I use git reset --hard fa120fa6b172f13b0535434caa93d838d739762c it says not a git repository or any of the parent directories

[cpu]

@GlitchStorm - Did you run that command from within the directory you git clone'd the Streisand repo? It sounds like you might have run it from the wrong directory.

I don't have any OSX hardware to try and reproduce this issue with. Long-term rolling back to fa120fa will be a poor workaround as you'll be missing out on all of the bug fixes & development work done on master.

I would be really interested in someone from the community affected by this bug trying to narrow down a reproduction case that I could run from Linux, or proposing a fix. Apologies to those affected!

[GlitchStorm]

@cpu that fixed the issue I had with that command not working, but I still get the same error in the end when I run streisand

[cpu]

There was a duplicate occurrence of the "Alert_cpu_threshold" error in https://github.com/jlund/streisand/issues/771 from a user on OSX with Ansible 2.3.1.0.

This issue is tagged status/help-wanted because there isn't anyone who has stepped up to own figuring out the root cause. Reiterating again that help is wanted!

[cmeid]

As described in https://github.com/ansible/ansible/issues/23873#issuecomment-296286207, adding the following lines:

        alert_cpu_threshold: 90
        alert_diskio_threshold: 10000
        alert_bwin_threshold: 10
        alert_bwout_threshold: 10
        alert_bwquota_threshold: 80

below

    ssh_pub_key: "{{ ssh_key.stdout }}"

to playbooks/roles/genesis-linode/tasks/main.yml solves the problem.

I haven't researched whether those settings are optimal, but adding them allows the build process to continue.

[witte-de-with]

thanks @cmeid !

[cpu]

@cmeid Thanks for suggesting that workaround. I haven't been able to reproduce this bug but I will try adding those lines to a branch and seeing if it still works for a system that doesn't exhibit this bug. If so then I think it makes sense to merge to master. If it breaks a system that doesn't exhibit this bug we'll have to do some deeper analysis to find a solution that works for the affected users without breaking new ones :-)

[cpu]

@ianbarton (Or someone else with an affected system) Can you try running the cpu-trial-linode-alert-fix branch of my fork: https://github.com/cpu/streisand/tree/cpu-trial-linode-alert-fix - I'm curious to see if setting only the alert_cpu_threshold that's directly referenced by the error is sufficient or if we need to set the diskio, bwin, bwout, and bwquota thresholds as well. Thanks!

[cpu]

I will try adding those lines to a branch and seeing if it still works for a system that doesn't exhibit this bug

I was able to confirm that adding the threshold variables caused no harm on my own system that was provisioning Linode instances without exhibiting the alert_cpu_threshold bug. :+1:

[cmeid]

I know that at the time I started trying with just the alert_cpu_threshold and then got the message about a different variable. I don't recall for certain whether I tried another single variable before inserting them all, but I believe I did try the next variable that I got a message about and then inserted the rest after two single tries.

[cpu]

@cmeid That's what I was afraid might happen :frowning_face: Perhaps I should just add the other parameters and not sink too much time into figuring out the minimal diff.

[witte-de-with]

@cpu @cmeid anything I can do to help keep this one moving?

[cpu]

@i-s-o-g-r-a-m Can you try running my cpu-trial-linode-alert-fix branch: https://github.com/cpu/streisand/tree/cpu-trial-linode-alert-fix and seeing if the problem persists? Per @cmeid's last comment I suspect it will still break because I only defined one alert. If you can confirm that I will try another commit and we can iterate. The primary blockers from my side are that I can't reproduce it and time :-)

[witte-de-with]

@cpu I tried your branch right now and got

TASK [genesis-linode : Create the server] **************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Alert_diskio_threshold must be between 0 and 100000."}
    to retry, use: --limit @/Users/andy/projects/streisand-fork/streisand/playbooks/linode.retry

[cpu]

@i-s-o-g-r-a-m Thanks for checking! Can you fetch that branch again and re-try? I just pushed 577b0caaff4353123200c6e7dda1dbb8b5c3a1aa which adds the other parameters @cmeid suggested. If that works on your system then I'll open a PR.

[witte-de-with]

@cpu I'll give it a shot and get back to you shortly -- thanks!

[witte-de-with]

@cpu no problems provisioning Linode with your branch -- LGTM 👍 thanks for putting in the work on this one!

[cpu]

@i-s-o-g-r-a-m Great! I put out a PR (https://github.com/jlund/streisand/pull/949) so we can get this fixed in master soon. :tada: :cake:

[witte-de-with]

@cpu awesome, thanks again!