complete provisioning scripts

[chadwhitacre]

Closes #224.

• [x] pick base Droplet Ubuntu image
• [x] configure inbound/outbound ports
• [ ] write provisioning scripts (in flight)
• [ ] figure out why Puma is not reading in the ENV variables correctly
• [ ] add command line arguments for dbuser, dbpass, and secrets rather than generating on the fly
• [x] add an on system startup script for Puma (completed here: cbf041e)

[chadwhitacre]

Alright. What am I supposed to do with this? Run it on an Ubuntu 14.x host?

[chadwhitacre]

[chadwhitacre]

What's missing on these scripts, @MatthewVita? How can I help?

[chadwhitacre]

Okay! I did some clean-up on the scripts to make the whole process easier to invoke, and now I'm running it against an Ubuntu 14.04 droplet for the first time.

[chadwhitacre]

http://askubuntu.com/questions/156650/apt-get-update-very-slow-stuck-at-waiting-for-headers

[chadwhitacre]

What's missing on these scripts, @MatthewVita? How can I help?

Just found this reply in private email:

Saw your comments on Github about wanting to helping with the provisioning stuff.

I'll follow up with you after work but the short answer is the learnpgh.org on Digital Ocean was just thrown together by hand for the Demo. It is temporary.

I'm currently finishing up the scripts in the "provisioning/src" area to be used on AWS EC2. If you want to help, get on the EC2 Free Tier and open up ports SSH/HTTP in security groups and follow the README.md in provisioning.

Basically the remaining work is:

• figure out why Puma is not reading in the ENV variables correctly (this is my biggest bottleneck... $APP_SECRET, $DATABASE_USER, etc aren't being picked up!)
• add a on system startup script for Puma
• add a Github WebHook that hits a custom script on our EC2 severs to deploy latest (I'm open for comments here.. please let me know if you want use another strategy/proper CI)

[chadwhitacre]

Failure!

Get:1 http://mirrors.digitalocean.com trusty/universe amd64 Packages [5,859 kB]
100% [Waiting for headers]
Err http://mirrors.digitalocean.com trusty/universe i386 Packages
  404  Not Found [IP: 192.241.164.26 80]
Err http://mirrors.digitalocean.com trusty/main i386 Packages
  404  Not Found [IP: 192.241.164.26 80]
Err http://mirrors.digitalocean.com trusty/universe Translation-en_US
  Bad header line [IP: 192.241.164.26 80]
Get:2 http://mirrors.digitalocean.com trusty/main Translation-en [943 kB]
Ign http://mirrors.digitalocean.com trusty/main Translation-en_US
Get:3 http://mirrors.digitalocean.com trusty/universe Translation-en [18.6 MB]
Fetched 25.4 MB in 6min 4s (69.9 kB/s)                              
W: Failed to fetch http://mirrors.digitalocean.com/ubuntu/dists/trusty/main/binary-i386/Packages  404  Not Found [IP: 192.241.164.26 80]

W: Failed to fetch http://mirrors.digitalocean.com/ubuntu/dists/trusty/universe/binary-i386/Packages  404  Not Found [IP: 192.241.164.26 80]

W: Failed to fetch http://mirrors.digitalocean.com/ubuntu/dists/trusty/universe/i18n/Translation-en_US  Bad header line [IP: 192.241.164.26 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.
root@deployment-test:~#

[chadwhitacre]

Basically the remaining work is:

@MatthewVita Cool. I've added those to the task list in this PR's description.

[chadwhitacre]

Error running as root on a clean droplet:

Fetched 902 kB in 7s (126 kB/s)                                                
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-3.13.0-61 linux-headers-3.13.0-61-generic
  linux-image-3.13.0-61-generic linux-image-extra-3.13.0-61-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  git git-man liberror-perl
Suggested packages:
  git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk
  gitweb git-arch git-bzr git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  git git-core git-man liberror-perl
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,347 kB of archives.
After this operation, 21.6 MB of additional disk space will be used.
Get:1 http://mirrors.digitalocean.com/ubuntu/ trusty/main liberror-perl all 0.17-1.1 [21.1 kB]
Get:2 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/main git-man all 1:1.9.1-1ubuntu0.1 [698 kB]
Get:3 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/main git amd64 1:1.9.1-1ubuntu0.1 [2,627 kB]
Get:4 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/main git-core all 1:1.9.1-1ubuntu0.1 [1,458 B]
Fetched 3,347 kB in 0s (8,613 kB/s)
Selecting previously unselected package liberror-perl.
(Reading database ... 116454 files and directories currently installed.)
Preparing to unpack .../liberror-perl_0.17-1.1_all.deb ...
Unpacking liberror-perl (0.17-1.1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../git-man_1%3a1.9.1-1ubuntu0.1_all.deb ...
Unpacking git-man (1:1.9.1-1ubuntu0.1) ...
Selecting previously unselected package git.
Preparing to unpack .../git_1%3a1.9.1-1ubuntu0.1_amd64.deb ...
Unpacking git (1:1.9.1-1ubuntu0.1) ...
Selecting previously unselected package git-core.
Preparing to unpack .../git-core_1%3a1.9.1-1ubuntu0.1_all.deb ...
Unpacking git-core (1:1.9.1-1ubuntu0.1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up liberror-perl (0.17-1.1) ...
Setting up git-man (1:1.9.1-1ubuntu0.1) ...
Setting up git (1:1.9.1-1ubuntu0.1) ...
Setting up git-core (1:1.9.1-1ubuntu0.1) ...
fatal: Could not change back to '/root': Permission denied

[chadwhitacre]

Okay! I seem to be able to get into the meat of the installation now, starting from a clean 14.04 droplet and running:

$ curl https://raw.githubusercontent.com/saxifrage/cityasacampus/deployment/provisioning/src/bootstrap.sh | bash

[chadwhitacre]

Looks like I'm compiling Ruby?

 #define PUSH_TAG() TH_PUSH_TAG(GET_THREAD())
                    ^
eval.c:99:5: note: in expansion of macro ‘PUSH_TAG’
     PUSH_TAG();
     ^
compiling load.c
In file included from load.c:9:0:
load.c: In function ‘rb_load_internal0’:
eval_intern.h:149:15: warning: ‘_th’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     th->state = 0;
               ^
eval_intern.h:123:23: note: ‘_th’ was declared here
   rb_thread_t * const _th = (th); \
                       ^
eval_intern.h:141:20: note: in expansion of macro ‘TH_PUSH_TAG’
 #define PUSH_TAG() TH_PUSH_TAG(GET_THREAD())
                    ^
load.c:604:5: note: in expansion of macro ‘PUSH_TAG’
     PUSH_TAG();
     ^
compiling proc.c
In file included from proc.c:12:0:
proc.c: In function ‘rb_method_call_with_block’:
eval_intern.h:149:15: warning: ‘_th’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     th->state = 0;
               ^
eval_intern.h:123:23: note: ‘_th’ was declared here
   rb_thread_t * const _th = (th); \
                       ^
eval_intern.h:141:20: note: in expansion of macro ‘TH_PUSH_TAG’
 #define PUSH_TAG() TH_PUSH_TAG(GET_THREAD())
                    ^
proc.c:1798:5: note: in expansion of macro ‘PUSH_TAG’
     PUSH_TAG();
     ^
proc.c:1818:16: warning: ‘data’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  defined_class = data->defined_class;
                ^
compiling file.c
compiling gc.c
compiling hash.c
compiling inits.c
compiling io.c

[chadwhitacre]

Looks like it ... worked?

   /etc/rc6.d/K20nginx
 Adding system startup for /etc/init.d/nginx ...
   /etc/rc0.d/K20nginx -> ../init.d/nginx
   /etc/rc1.d/K20nginx -> ../init.d/nginx
   /etc/rc6.d/K20nginx -> ../init.d/nginx
   /etc/rc2.d/S20nginx -> ../init.d/nginx
   /etc/rc3.d/S20nginx -> ../init.d/nginx
   /etc/rc4.d/S20nginx -> ../init.d/nginx
   /etc/rc5.d/S20nginx -> ../init.d/nginx
 * Restarting nginx nginx
   ...done.
Removing rails (4.2.4)
Removing arel (6.0.3)
Removing mini_portile2 (2.0.0.rc2)
Removing loofah (2.1.0.rc1)
Removing test-unit (2.1.4.0)
Removing rack (1.6.4)
Removing actionpack (4.2.4)
Removing activerecord (4.2.4)
Removing activesupport (4.2.4)
Removing globalid (0.3.6)
Removing sprockets (3.4.0)
Removing activemodel (4.2.4)
Removing railties (4.2.4)
Removing activejob (4.2.4)
Removing mime-types (2.99)
Removing sprockets-rails (3.0.0.beta2)
Removing nokogiri (1.6.7.rc4)
Removing actionview (4.2.4)
Removing crass (1.0.2)
Removing minitest (5.8.3)
Removing rdoc (4.1.0)
Removing rake (10.1.0)
Removing actionmailer (4.2.4)
Removing rails-dom-testing (1.0.7)
[32395] Puma starting in cluster mode...
[32395] * Version 2.11.1 (ruby 2.1.4-p265), codename: Intrepid Squirrel
[32395] * Min threads: 1, max threads: 6
[32395] * Environment: production
[32395] * Process workers: 1
[32395] * Phased restart available
[32395] * Listening on unix:///home/caac/cityasacampus/shared/sockets/puma.sock
[32395] * Daemonizing...
root@test:~#

[chadwhitacre]

figure out why Puma is not reading in the ENV variables correctly (this is my biggest bottleneck... $APP_SECRET, $DATABASE_USER, etc aren't being picked up!)

What is a symptom of Puma not reading in the ENV variables correctly? How can I reproduce the bug?

[chadwhitacre]

O.O

!m @MatthewVita

[chadwhitacre]

Maybe this is a symptom?

[chadwhitacre]

Where does puma log?

[chadwhitacre]

stdout_redirect "#{shared_dir}/log/puma.stdout.log", "#{shared_dir}/log/puma.stderr.log", true

https://github.com/saxifrage/cityasacampus/blob/caaa47be7147ed5474944c9f91ccb807fe0a6fd3/provisioning/src/puma_config#L9

[chadwhitacre]

Seems like two of these are redundant:

/home/caac/cityasacampus/log/
/home/caac/cityasacampus/shared/log/
/home/caac/cityasacampus/shared/logs/

Puma logs to the second.

[chadwhitacre]

In puma.stderr.log:

=== puma startup: 2015-11-23 14:48:38 -0500 ===
/usr/local/lib/ruby/gems/2.1.0/gems/devise-3.5.1/lib/devise/rails/routes.rb:480:in `raise_no_secret_key': Devise.secret_key was not set. Please add the following to your Devise initializer: (RuntimeError)

  config.secret_key = '[]'

[chadwhitacre]

Here's the diff where we started trying to pull APP_SECRET from ENV: https://github.com/saxifrage/cityasacampus/commit/0c64f90cbf9ca573387871bdc1b50948cae5d17c#diff-a8a4112d0760c0fd11f691c6a5fb22adL9

[chadwhitacre]

I don't see anything in puma.sh that suggests how we might be influencing the process environment of pumactl. Does pumactl have any facilities for this? Any defaults?

[chadwhitacre]

how we might be influencing the process environment

Here it is, in project.sh.

[chadwhitacre]

root@test:/home/caac/cityasacampus# tail /etc/profile
      . $i
    fi
  done
  unset i
fi
export RAILS_ENV=production
export APP_DATABASE_USER=
export APP_DATABASE_PASS=
export APP_SECRET=
export APP_TOKEN=

[chadwhitacre]

The interactive prompting I added in https://github.com/saxifrage/cityasacampus/commit/1feeffa9dcf9c8120b88d0bd4fee2052d9cfb6db fails under bootstrap.sh.

[chadwhitacre]

... though it sounds like the /etc/profile mechanism wasn't working as expected to begin with.

[chadwhitacre]

Hypothesis: The changes to /env/profile aren't being sourced into the shell session from which pumactl is initially run.

Test: Running pumactl from a shell that has the environment configured properly should work.

[chadwhitacre]

I am unable to start puma:

root@test:/home/caac/cityasacampus/shared/pids# pumactl -P puma.pid start
Puma starting in single mode...
* Version 2.11.1 (ruby 2.1.4-p265), codename: Intrepid Squirrel
* Min threads: 0, max threads: 16
* Environment: development
ERROR: No application configured, nothing to run

[chadwhitacre]

pumactl expects to be in the root directory of a Rails application:

root@test:/home/caac/cityasacampus# pumactl -P shared/pids/puma.pid start
[9006] Puma starting in cluster mode...
[9006] * Version 2.11.1 (ruby 2.1.4-p265), codename: Intrepid Squirrel
[9006] * Min threads: 1, max threads: 6
[9006] * Environment: production
[9006] * Process workers: 1
[9006] * Phased restart available
[9006] * Listening on unix:///home/caac/cityasacampus/shared/sockets/puma.sock
[9006] * Daemonizing...

[chadwhitacre]

Progress! :dancer:

504 → 502 → 302 → 500

[chadwhitacre]

Hypothesis: The changes to /env/profile aren't being sourced into the shell session from which pumactl is initially run.

Looks like we try, anyway.

[chadwhitacre]

Looks like the 500 is PG::UndefinedTable: ERROR: relation "users" does not exist. If I didn't have APP_DATABASE_* set properly, I guess the rake db calls in project.sh didn't take?

[chadwhitacre]

I ran bootstrap.sh on a clean droplet, and /etc/profile now has secrets in it. However, I am still seeing the raise_no_secret_key error in puma.stderr.log. Furthermore, fwiw, env in the parent shell where I ran bootstrap.sh does not have the envvars set.

[chadwhitacre]

If an environment is specified, either via the -e and --environment flags, or through the RACK_ENV environment variable, the default file location will be config/puma/environment_name.rb.

[chadwhitacre]

Why ENV['FOG_DIRECTORY'] is nil after server reboot? The answer is simple - nginx or another web server (I don’t know which one you use, but I prefer using nginx) starts before evaluating ~/.bashrc and even /etc/environment.

http://railsguides.net/how-to-define-environment-variables-in-rails/

[chadwhitacre]

The author there recommends moving config to YAML and writing into ENV inside Ruby. The top comment points to Foreman.

[chadwhitacre]

This DigitalOcean tutorial says to us rbenv-vars.

[chadwhitacre]

If you have any Ruby web apps deployed with Puma (on Linux), you may have run in to the problem whereby there is no obvious way of bringing in Environment Variables before the Apps spin up.

http://www.vpschef.com/2014/08/hack-to-environment-variables-in-to-the-puma-jungle/

There the solution is to "Edit the file /usr/local/bin/run-puma." O.o

[chadwhitacre]

If you're relying on environment variables, it can lead to frustration and confusion when the process works when you start in manually, but not when cron or monit does.

Instead of environment variables, we recommend that you either use a YAML configuration file or a shared data source.

https://support.cloud.engineyard.com/hc/en-us/articles/205407508-Environment-Variables-and-Why-You-Shouldn-t-Use-Them

Wherein EngineYard accuses "some PaaS providers" *cough*Heroku*cough* of using environment variables as "a hack to get around their file system limitations." :D

[chadwhitacre]

Heroku has you denote puma directly in your Procfile, bypassing pumactl.

[chadwhitacre]

Alright, that's all I got for now. Back over to you, @MatthewVita. :-)

[MatthewVita]

@whit537 see comment here: https://github.com/saxifrage/cityasacampus/issues/224#issuecomment-159133017

[chadwhitacre]

Closing in favor of #354.