ssh connection timeout on vagrant up

$ vagrant up st2 --provider=virtualbox

Bringing machine 'st2' up with 'virtualbox' provider...

==> st2: Importing base box 'stackstorm/st2express'...
==> st2: Matching MAC address for NAT networking...
==> st2: Checking if box 'stackstorm/st2express' is up to date...
==> st2: Setting the name of the VM: st2workroom_st2_1447654085174_26976
==> st2: Clearing any previously set network interfaces...
==> st2: Preparing network interfaces based on configuration...
    st2: Adapter 1: nat
    st2: Adapter 2: hostonly
==> st2: Forwarding ports...
    st2: 22 => 2222 (adapter 1)
==> st2: Running 'pre-boot' VM customizations...
==> st2: Booting VM...
==> st2: Waiting for machine to boot. This may take a few minutes...
    st2: SSH address: 127.0.0.1:2222
    st2: SSH username: vagrant
    st2: SSH auth method: private key
    st2: Warning: Connection timeout. Retrying...
    st2: Warning: Connection timeout. Retrying...
    st2: Warning: Connection timeout. Retrying...
    st2: Warning: Connection timeout. Retrying...
    st2: Warning: Connection timeout. Retrying...

Vagrant version: 1.7.4 VirtualBox version: 5.0.10r104061 Host: Ubuntu 14.04

Is there any change that needs to be done to Vagrantfile to enable the ssh connectivity?

@mukulashokjoshi Sorry you're having issues!

Typically, you should have to do nothing except type the command you did, and it should come up without problem.

I've tried to replicate your issue unsuccessfully. Let's try a few things!

What version of the stackstorm/st2express VM do you currently have installed? You can find this with vagrant box list.
- There is a known issue with v60, and it was revoked. If you have v60, go ahead and delete that box and use v59
It might be that for some very odd reason, the node is waiting on input. I have seen this before with other boxes, so it's worth a shot attempting to take a look here as well. Do the following:

While your VM is booting, in a separate terminal window, just find out the id of the running machine.

vboxmanage list runningvms

This will result in something like this:

"projects_1234567890" {5cxxxx-cxxx-4xxx-8xxx-5xxxxxxxxxx}

Quite often, the VM is simply waiting for you to select an option in the bootloader. You can send the appropriate keycode (in the case, Enter) to the vm with controlvm:

vboxmanage controlvm projects_1234567890 keyboardputscancode 1c

If neither of these work, drop a note in here with the specs of the host machine, and we'll keep digging!

@jfryman Thanks for taking time out to reply to this

Suggestion 1: output of "vagrant box list" is - stackstorm/st2express (virtualbox, 59) Suggestion 2: No luck

System information from "inxi -Fxz" in "sysinfo.txt" attached sysinfo.txt

Debug1.txt from running "vagrant up st2 --provider=virtualbox --debug 2>&1 | tee debug1.log" attached debug1.txt

Debug2.txt from running "vagrant ssh st2 --debug 2>&1 | tee debug2.log" attached debug2.txt

I have also generated private SSH keys by following instructions from this URL: http://www.productiveminds.com/blog/vagrant-ssh-authentication-how-to-successfully-login-into-vm-box-using-vagrant-up/. But in absence of complete understanding of the setup, have not been able to make the changes for the "private key path". Adding anything like "config.vm." or "config.ssh." in Vagrantfile results in an error. Some of the ssh stuff is there in files like "st2-atlas.json" and "workroom.yaml". If possible, some pointers on how to make changes would be useful.

Thanks & Regards

@mukulashokjoshi I spent a considerable amount of time yesterday trying to find something that is causing issues. I'm still digging, but I do want to get you unblocked.

First, where do you add things:

https://github.com/StackStorm/st2workroom/blob/master/Vagrantfile#L139

Anywhere inside the scope of this block is safe to add attributes to the machine. Try seeing if you can increase the boot timeout, or even the SSH key management. The private key on the box is in fact the master key shipped with most boxes, so I'm not sure if that is the underlying issue. However, would love to hear your findings.

I might also add the following to the Vagrantfile just to get a GUI and see what's going on. Insert a line around https://github.com/StackStorm/st2workroom/blob/master/Vagrantfile#L156, and add the following inside the n.vm.provider 'virtualbox' do |virtualbox| block.

  virtualbox.gui = true

Finally, another thing to consider is to just spin up a fresh VM in your favorite way (ignore StackStorm st2workroom) and instead use the installer script. Take a look at http://docs.stackstorm.com/install/all_in_one.html#bring-your-own-box.

Let us know how it goes, we'll get you up and running!

@jfryman Thanks for your time on this

I have started the "all in one" installation (have obfuscated the actual proxy URLs below):

Initially did not provide the HTTPS proxy for the root user. There is a HTTPS proxy configured for a normal user, but the root does not pick it up. Tried to use various mechanism to set the proxy for root (/etc/environment, /root/.profile, /etc/bash.bashrc with sudo bash), but these did not work. So passed the proxy in the command itself:

curl -sSL http://stackstorm.com/install.sh | sudo env https_proxy='https://..' sh

This solved the initial issue (not able to access Github over HTTPS), but failed later on trying to use HTTPS proxy in place of required HTTP proxy:

Error: Could not set 'present' on ensure: Non-HTTP proxy URI: https://... at 88:/opt/puppet/shared/apt/manifests/key.pp
Error: Could not set 'present' on ensure: Non-HTTP proxy URI: https://... at 88:/opt/puppet/shared/apt/manifests/key.pp
Wrapped exception:
Non-HTTP proxy URI: https://...
Error: /Stage[main]/Nodejs::Repo::Nodesource::Apt/Apt::Source[nodesource]/Apt::Key[Add key: 9FD3B784BC1C6FC31A8A0A1C1655A0AB68576280 from Apt::Source nodesource]/Apt_key[Add key: 9FD3B784BC1C6FC31A8A0A1C1655A0AB68576280 from Apt::Source nodesource]/ensure: change from absent to present failed: Could not set 'present' on ensure: Non-HTTP proxy URI: https://... at 88:/opt/puppet/shared/apt/manifests/key.pp

So then added the HTTP proxy also to the command and did a rerun:

curl -sSL http://stackstorm.com/install.sh | sudo env http_proxy='http://...' https_proxy='https://...' sh

But unfortunately, it fails with the same error. Is the proxy URL stored somewhere?

Is there a way to do a complete clean-up and re-start?

Tried the "update-system" too (with "sudo env http_proxy='http://...' https_proxy='https://...' sh"), but this also fails with the same error

Did a grep in the /opt to see if the proxy gets stored somewhere in the install, but could not find any trace

Thanks

@mukulashokjoshi OK! Well, good news is that we both know what causes the issue with the proxy and apt, have a fix underway, and have a workaround right now.

Take a look at https://stackstorm.reamaze.com/kb/installation/connection-reset-by-peer-trying-to-download-openpgp-keys (we're beta testing some KB tools, so this may move... but the instructions are solid). This should get you going, and I'll get a fix in shortly.

@jfryman Thanks

Update on the "all in one" installation:

As per your recommendation, used the workaround in the #269 issue and manually updated the keys. There was no need to enable the TCP port, as use of the port 80 sufficed as shown below:

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --keyserver-options http_proxy='http://HTTP_PROXY' --recv-keys KEY_ID

But there was another error which resulted in failure related to network timed out:

Notice: /Stage[main]/Profile::Rabbitmq/Docker::Image[rabbitmq]/Exec[docker pull rabbitmq]/returns: Pulling repository docker.io/library/rabbitmq
Notice: /Stage[main]/Profile::Rabbitmq/Docker::Image[rabbitmq]/Exec[docker pull rabbitmq]/returns: Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/rabbitmq/images. You may want to check your internet connection or if you are behind a proxy.
Error: docker pull rabbitmq returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Rabbitmq/Docker::Image[rabbitmq]/Exec[docker pull rabbitmq]/returns: change from notrun to 0 failed: docker pull rabbitmq returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/Exec[/bin/sh /etc/init.d/docker-rabbitmq stop]: Dependency Exec[docker pull rabbitmq] has failures: true
Warning: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/Exec[/bin/sh /etc/init.d/docker-rabbitmq stop]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/File[/var/run/docker-rabbitmq.cid]: Dependency Exec[docker pull rabbitmq] has failures: true
Warning: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/File[/var/run/docker-rabbitmq.cid]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/File[/etc/init.d/docker-rabbitmq]: Dependency Exec[docker pull rabbitmq] has failures: true
Warning: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/File[/etc/init.d/docker-rabbitmq]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/Service[docker-rabbitmq]: Dependency Exec[docker pull rabbitmq] has failures: true
Warning: /Stage[main]/Profile::Rabbitmq/Docker::Run[rabbitmq]/Service[docker-rabbitmq]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Mongodb/Docker::Image[mongo]/Exec[docker pull mongo:2.4.14]/returns: Pulling repository docker.io/library/mongo
Notice: /Stage[main]/Profile::Mongodb/Docker::Image[mongo]/Exec[docker pull mongo:2.4.14]/returns: Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/mongo/images. You may want to check your internet connection or if you are behind a proxy.
Error: docker pull mongo:2.4.14 returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Mongodb/Docker::Image[mongo]/Exec[docker pull mongo:2.4.14]/returns: change from notrun to 0 failed: docker pull mongo:2.4.14 returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Mongodb/Exec[create mongodb data container]: Dependency Exec[docker pull mongo:2.4.14] has failures: true
Warning: /Stage[main]/Profile::Mongodb/Exec[create mongodb data container]: Skipping because of failed dependencies

Update on the ssh connection timeout:

As part of various workarounds to overcome the ssh connection timeout, one is related to enabling Virtualisation in BIOS. Refer: http://stackoverflow.com/questions/22575261/vagrant-stuck-connection-timeout-retrying & https://github.com/Varying-Vagrant-Vagrants/VVV/issues/375

So a separate box, recreated the issue and then checked the Virtualisation (Intel) in the BIOS (UEFI). Here it was observed that Virtualisation was not enabled. After enabling Virtualisation, re-ran the vagrant up st2 and script was able to connect to VM via SSH and complete the provisioning. There were still some timeouts while updating some of the 'packs', but this did not result in a fatal error and finally the ST2 OK message was displayed. This box has Linux Mint 17.2 which has Ubuntu 14.04 underlying.

Will have to see how to enable Virtualisation for the original box where we saw the error, which only has remote SSH access

@mukulashokjoshi We've identified several parts of the installer that needs additional :heart: with regards to Proxies. We're currently taking a look, and trying to get a better story here.

I will keep this issue updated as we make additional movement. Apologies for the inconveniences, but we'll get you taken care of!

@jfryman Thanks

Update-2 on the "all in one" installation:

We have been doing some workarounds so as to see if we can move forward with the install.

Workaround-1 : Enabling the docker load

We decided to “docker pull, save" on a separate box, then FTPed the docker tar image to remote server and then did a "docker load" of the tar files. This Stackoverflow comment was useful: http://stackoverflow.com/questions/22381442/pulling-docker-images

This was followed for rabbitmq & mongo:2.4.14, for which there were "network timed out" errors

Re-ran the ST2 install with "/opt/puppet/script/bootstrap-st2" which resulted in a another error:

Error: Could not start Service[docker-mongo]: Execution of '/etc/init.d/docker-mongo start' returned 1: Error: No such image or container: mongo
Starting docker-mongo:  Unable to find image 'mongo:latest' locally
Pulling repository docker.io/library/mongo
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/mongo/images. You may want to check your internet connection or if you are behind a proxy.

Initially error was for "mongo:2.4.14" & "rabbitmq", when these were loaded, then error was for "mongo:latest". Loaded the "mongo:latest" docker image following procedure as above.

Workaround-2 : Enabling the docker proxy

The proxy to be used by docker was "hard coded" in this file:

/opt/puppet/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/garethr-docker/4.1.1/docker/templates/etc/default/docker.erb

Re-ran the ST2 install, which resulted in some more errors:

Error: st2 run packs.install packs=libcloud    returned 1 instead of one of [0]
Error: /Stage[main]/St2::Packs/St2::Pack[libcloud]/Exec[install-st2-pack-libcloud]/returns: change from notrun to 0 failed: st2 run packs.install packs=libcloud    returned 1 instead of one of [0]
Notice: /Stage[main]/St2::Packs/St2::Pack[jira]/Exec[install-st2-pack-jira]/returns: 2015-11-30 10:56:57,490  WARNING - Auth API server is not available, skipping authentication.
Notice: /Stage[main]/St2::Packs/St2::Pack[jira]/Exec[install-st2-pack-jira]/returns: ERROR: HTTPSConnectionPool(host='10.56.183.27', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))
Error: st2 run packs.install packs=jira    returned 1 instead of one of [0]
Error: /Stage[main]/St2::Packs/St2::Pack[jira]/Exec[install-st2-pack-jira]/returns: change from notrun to 0 failed: st2 run packs.install packs=jira    returned 1 instead of one of [0]
Notice: /Stage[main]/St2::Packs/St2::Pack[openstack]/Exec[install-st2-pack-openstack]/returns: 2015-11-30 10:57:15,235  WARNING - Auth API server is not available, skipping authentication.
Notice: /Stage[main]/St2::Packs/St2::Pack[openstack]/Exec[install-st2-pack-openstack]/returns: ERROR: HTTPSConnectionPool(host='10.56.183.27', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))
Error: st2 run packs.install packs=openstack    returned 1 instead of one of [0]
Error: /Stage[main]/St2::Packs/St2::Pack[openstack]/Exec[install-st2-pack-openstack]/returns: change from notrun to 0 failed: st2 run packs.install packs=openstack    returned 1 instead of one of [0]
Notice: /Stage[main]/St2::Packs/St2::Pack[aws]/Exec[install-st2-pack-aws]/returns: 2015-11-30 10:57:32,867  WARNING - Auth API server is not available, skipping authentication.
Notice: /Stage[main]/St2::Packs/St2::Pack[aws]/Exec[install-st2-pack-aws]/returns: ERROR: HTTPSConnectionPool(host='10.56.183.27', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))
Error: st2 run packs.install packs=aws    returned 1 instead of one of [0]
Error: /Stage[main]/St2::Packs/St2::Pack[aws]/Exec[install-st2-pack-aws]/returns: change from notrun to 0 failed: st2 run packs.install packs=aws    returned 1 instead of one of [0]
Notice: /Stage[main]/St2::Packs/St2::Pack[twitter]/Exec[install-st2-pack-twitter]/returns: 2015-11-30 10:57:50,492  WARNING - Auth API server is not available, skipping authentication.
Notice: /Stage[main]/St2::Packs/St2::Pack[twitter]/Exec[install-st2-pack-twitter]/returns: ERROR: HTTPSConnectionPool(host='10.56.183.27', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))
Error: st2 run packs.install packs=twitter    returned 1 instead of one of [0]
Error: /Stage[main]/St2::Packs/St2::Pack[twitter]/Exec[install-st2-pack-twitter]/returns: change from notrun to 0 failed: st2 run packs.install packs=twitter    returned 1 instead of one of [0]

And auth related:

2015-11-30 10:58:47,045  WARNING - Auth API server is not available, skipping authentication.
ERROR: HTTPSConnectionPool(host=' . . . ', port=9101): Max retries exceeded with url: /actions/core.local (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))

But were able to get the ST2 OK message

The setup was then accessed at: https://BOX_IP/setup

The basic setup was done and at the end of the setup, the installation completion was triggered which resulted in further errors (there is github error related to proxy amongst others) as noted in this log: Install_Errors.txt

On accessing the https://BOX_IP, also got "invalid or missing credentials" when trying to login as "admin" which was configured during setup process

@mukulashokjoshi It looks like you're so close!

First, I really appreciate the investigation you're doing. We have a proxy server lab now setup on our end, so we'll officially certify support with Proxies very soon. Your data here is most helpful! :bow:

In the short term, I think you have the bits you need... and now you've run into this error.

https://docs.stackstorm.com/install/all_in_one.html#errors-on-connection

Try adding the correct ethernet adapter to the answers.yaml file as indicated in the doc. Also, be careful when copy/pasting the text from the help document, as it has UTF-8 quotation marks, and that does not play well with Puppet.

Let us know how it goes!

@jfryman Thanks

Will go through the new details mentioned by you

Update-3 on the "all in one" installation:

Executing "update-system" resulted in some certificate errors:

Error: /Stage[main]/Profile::St2server/Vcsrepo[/etc/st2installer]: Could not evaluate: Execution of '/usr/bin/git fetch origin' returned 128: fatal: unable to access 'https://github.com/stackstorm/st2installer/': Problem with the SSL CA cert (path? access rights?)
Error: /Stage[main]/St2::Profile::Mistral/Vcsrepo[/opt/openstack/mistral]: Could not evaluate: Execution of '/usr/bin/git fetch origin' returned 128: fatal: unable to access 'https://github.com/StackStorm/mistral.git/': Problem with the SSL CA cert (path? access rights?)
Error: /Stage[main]/St2::Profile::Mistral/Vcsrepo[/etc/mistral/actions/st2mistral]: Could not evaluate: Execution of '/usr/bin/git fetch origin' returned 128: fatal: unable to access 'https://github.com/StackStorm/st2mistral.git/': Problem with the SSL CA cert (path? access rights?)
Notice: /Stage[main]/Profile::St2server/Users[root_cli]/Exec[Setting root_cli password]/returns: executed successfully
Error: /Stage[main]/Profile::St2server/Ca_cert::Ca[stackstorm-ca]/File[stackstorm-ca.crt]: Could not evaluate: Could not retrieve file metadata for : Cannot find file: Invalid mount ''
Notice: /Stage[main]/Ca_cert::Update/Exec[ca_cert_update]: Dependency File[stackstorm-ca.crt] has failures: true
Warning: /Stage[main]/Ca_cert::Update/Exec[ca_cert_update]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::St2server/Ca_cert::Ca[stackstorm-ca]/Anchor[ca_cert::ca::stackstorm-ca]: Dependency File[stackstorm-ca.crt] has failures: true
Warning: /Stage[main]/Profile::St2server/Ca_cert::Ca[stackstorm-ca]/Anchor[ca_cert::ca::stackstorm-ca]: Skipping because of failed dependencies

Workaround-1: Enabling the ST2 services

By default it is observed that the services are not running. So we enabled the services with st2ctl:

usage: st2ctl {restart-component}
positional arguments:
  component           Name of the st2 service to restart.
                      st2actionrunner st2api st2auth st2sensorcontainer st2rulesengine st2web mistral st2resultstracker st2notifier

Workaround-2: Enabling the CA certificates

Some details on certificates are mentioned here: https://github.com/StackStorm/st2installer/issues/29

So the certificate from https://BOX_IP/ssl/st2_root_ca.cer was downloaded and installed into the Ubuntu certificatte store as per this URL: http://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu

Not sure if this is correct.

Workaround-3: Enabling the API URL

If the hostname is not provided, then by default the hostname is taken from the IP address as ip-NET1-NET2-NET3-HOST

The api_url in the st2.conf was changed to:

# Base URL to the API endpoint excluding the version (e.g. http://myhost.net:9101/)
api_url =https://ip-NET1-NET2-NET3-HOST:9101

After this certificate errors were not observed

Workaround-4: Enabling npm proxies

Some npm errors were observed related to proxies, so npm proxies were enabled with these:

npm config set proxy http://PROXY_IP:PORT
npm config set https-proxy https://PROXY_IP:PORT

Workaround-5: Enabling /setup to run multiple times

Since we were doing changes cyclically, we need to run https://BOX_IP/setup multiple times. But it was observed that this was not happening. It was observed that the install logs were stored in /tmp/st2installer.log and there was a related lock file /tmp/st2installer_lock. Renaming these, enabled the /setup to be run again

Workaround-6: Enabling authentication

By default the authentication is set to pam in the st2.conf as follows:

[auth]
host = 0.0.0.0
port = 9100
use_ssl = False
debug = True
enable = True
logging = /etc/st2auth/syslog.conf

mode = standalone

# Note: Settings bellow are only used in "standalone" mode
backend = pam
backend_kwargs =

This was changed to flat_file as follows:

[auth]
host = 0.0.0.0
port = 9100
use_ssl = False
debug = True
enable = True
logging = /etc/st2auth/syslog.conf

mode = standalone

# Note: Settings bellow are only used in "standalone" mode
backend = flat_file
backend_kwargs = {"file_path": "/etc/st2/.htpasswd"}

The password for the admin user was generated as follows:

htpasswd /etc/st2/.htpasswd <username> <password>

With this change the authentication worked and we were able to access the StackStorm HISTORY, ACTION and RULES!

Using Mongo DB for authentication

We came across this module: https://github.com/StackStorm/st2-auth-backend-mongodb/blob/master/README.md

This enables use of Mongo DB for authentication

It will be useful to get some pointers on how to integrate this module into the main build

Also it will be useful to some pointers on how to do the build for complete ST2

Somewhere have read that ST2 can be used to do build for ST2. Again will be useful to get some leads on this.

Errors while installing packs

[mNotice: /Stage[main]/St2::Packs/St2::Pack[hubot]/Exec[install-st2-pack-hubot]/returns: ERROR: HTTPSConnectionPool(host='BOX_IP', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))[0m
[1;31mError: st2 run packs.install packs=hubot    returned 1 instead of one of [0][0m
[1;31mError: /Stage[main]/St2::Packs/St2::Pack[hubot]/Exec[install-st2-pack-hubot]/returns: change from notrun to 0 failed: st2 run packs.install packs=hubot    returned 1 instead of one of [0][0m
[mNotice: /Stage[main]/St2::Packs/St2::Pack[hubot]/File[/opt/stackstorm/packs/hubot/config.yaml]: Dependency Exec[install-st2-pack-hubot] has failures: true[0m
[1;33mWarning: /Stage[main]/St2::Packs/St2::Pack[hubot]/File[/opt/stackstorm/packs/hubot/config.yaml]: Skipping because of failed dependencies[0m
[mNotice: /Stage[main]/St2::Packs/St2::Pack[st2]/Exec[install-st2-pack-st2]/returns: ERROR: HTTPSConnectionPool(host='BOX_IP', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))[0m
[1;31mError: st2 run packs.install packs=st2    returned 1 instead of one of [0][0m
[1;31mError: /Stage[main]/St2::Packs/St2::Pack[st2]/Exec[install-st2-pack-st2]/returns: change from notrun to 0 failed: st2 run packs.install packs=st2    returned 1 instead of one of [0][0m
[mNotice: /Stage[main]/St2::Packs/St2::Pack[puppet]/Exec[install-st2-pack-puppet]/returns: ERROR: HTTPSConnectionPool(host='BOX_IP', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))[0m
[1;31mError: st2 run packs.install packs=puppet    returned 1 instead of one of [0][0m
[1;31mError: /Stage[main]/St2::Packs/St2::Pack[puppet]/Exec[install-st2-pack-puppet]/returns: change from notrun to 0 failed: st2 run packs.install packs=puppet    returned 1 instead of one of [0][0m
[mNotice: /Stage[main]/St2::Packs/St2::Pack[github]/Exec[install-st2-pack-github]/returns: ERROR: HTTPSConnectionPool(host='BOX_IP', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))[0m
[1;31mError: st2 run packs.install packs=github    returned 1 instead of one of [0][0m
[1;31mError: /Stage[main]/St2::Packs/St2::Pack[github]/Exec[install-st2-pack-github]/returns: change from notrun to 0 failed: st2 run packs.install packs=github    returned 1 instead of one of [0][0m
[mNotice: /Stage[main]/St2::Packs/St2::Pack[victorops]/Exec[install-st2-pack-victorops]/returns: ERROR: HTTPSConnectionPool(host='BOX_IP', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))[0m
[1;31mError: st2 run packs.install packs=victorops    returned 1 instead of one of [0][0m
[1;31mError: /Stage[main]/St2::Packs/St2::Pack[victorops]/Exec[install-st2-pack-victorops]/returns: change from notrun to 0 failed: st2 run packs.install packs=victorops    returned 1 instead of one of [0][0m
[mNotice: /Stage[main]/St2::Packs/St2::Pack[sensu]/Exec[install-st2-pack-sensu]/returns: ERROR: HTTPSConnectionPool(host='BOX_IP', port=9101): Max retries exceeded with url: /actions/packs.install (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Tunnel or SSL Forbidden',)))[0m
[1;31mError: st2 run packs.install packs=sensu    returned 1 instead of one of [0][0m
[1;31mError: /Stage[main]/St2::Packs/St2::Pack[sensu]/Exec[install-st2-pack-sensu]/returns: change from notrun to 0 failed: st2 run packs.install packs=sensu    returned 1 instead of one of [0][0m

Any pointers on adding packs from st2contrib will also be useful

@jfryman

Update-4 on the "all in one" installation:

Workaround-1: Enabling packs install

In light of the issues faced with the packs install, which times out while trying connect to proxy, we tried to independently load the Git repo and then run the packs install. So a git clone was done and then tried to install the pack from the local git repo:

st2  run packs.install packs=hubot repo_url=/root/st2contrib.git

This did not work completely

So tried to do some more variations with the st2 run command:

st2  run -e packs.setup_virtualenv packs=hubot

This seems to work consistently and packs can be installed with the pack being downloaded from the net. The -e param seems to pick up the environment details for the proxy to work

Now we have some of the basic packs installed

We will try to do enable some automations

Ofcourse knowledge of complete build will be useful going forward

Thanks

@jfryman

The output of st2ctl status is:

##### st2 components status #####
st2actionrunner PID: 22221
st2actionrunner PID: 22222
st2actionrunner PID: 22223
st2actionrunner PID: 22224
st2actionrunner PID: 22225
st2actionrunner PID: 22226
st2actionrunner PID: 22227
st2actionrunner PID: 22228
st2actionrunner PID: 22229
st2api PID: 620
st2api PID: 22238
st2auth PID: 22245
st2sensorcontainer PID: 22259
st2rulesengine PID: 22270
st2web is not running.
mistral PID: 22289
mistral PID: 22291
st2resultstracker PID: 22317
st2notifier PID: 22330

And output of st2ctl stop is:

stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.510" (uid=1000 pid=29830 comm="stop st2actionrunner ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.512" (uid=1000 pid=29836 comm="stop st2api ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.514" (uid=1000 pid=29842 comm="stop st2auth ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.516" (uid=1000 pid=29848 comm="stop st2sensorcontainer ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.518" (uid=1000 pid=29854 comm="stop st2rulesengine ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.520" (uid=1000 pid=29860 comm="stop st2web ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.522" (uid=1000 pid=29866 comm="stop mistral ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.524" (uid=1000 pid=29872 comm="stop st2resultstracker ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.526" (uid=1000 pid=29878 comm="stop st2notifier ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")

Any clues as to why the st2ctl stop call is rejected?

Thanks

@mukulashokjoshi are you running that command as root or as a user that has root privileges? I have seen that error when running as an unprivileged user.

@jfryman

Thanks for the clarification

My mistake

sudo st2ctl stop works

##### st2 components status #####
st2actionrunner is not running.
st2api is not running.
st2auth is not running.
st2sensorcontainer is not running.
st2rulesengine is not running.
st2web is not running.
mistral is not running.
st2resultstracker is not running.
st2notifier is not running.

StorminStanley / st2workroom