Patterns for multiple environments / help me debug my Rails setup / rubber duckie session

Problem

As a dev, I am looking for a standard technique for hosting a web application in production and staging environments. There seem to be multiple ways to approach this problem, but this seems like a process that need not (and should not) be reinvented for each app.

I'm currently struggling through this for a Rails app, but I think the solution here could be framework/language-agnostic.

Current Approach (and down the rabbit hole)

Currently (for Micropurchase), we have an app in the production space of our CF org. We have a pretty standard Ruby buildpack, a psql-shared service, and deploys integrated into Travis-CI.

To get a staging site up, I added a staging space to our CF org and deployed using cf push -n micropurchase-staging. Here's where things get murky:

➜  micropurchase git:(master) ✗ cf push -n micropurchase-staging 
[... cf buildpack stuff ...]
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 failing
FAILED
Start unsuccessful

On to the logs!:

2015-12-21T06:48:25.36-0500 [App/0]      ERR PG::InsufficientPrivilege: ERROR:  permission denied to create database

(full logs here)

So this is weird and difficult to understand.

The Next Approach

Rather than specify the -n flag, lets try using a separate buildpack just for staging (manifest-staging.yml):

applications:
- name: micropurchase-staging
  memory: 128M
  instances: 1
  domain: 18f.gov
  hostname: micropurchase-staging
  buildpack: https://github.com/ddollar/heroku-buildpack-multi.git
  services:
    - micropurchase-psql

cf push -f manifest-staging.yml
[... cf buildpack stuff ...]
1 of 1 instances running

App started

OK

App micropurchase-staging was started using this command `bundle exec rails server -p $PORT`

Showing health and status for app micropurchase-staging in org agile-bpa / space staging as alan.delevie@gsa.gov...
OK

requested state: started
instances: 1/1
usage: 128M x 1 instances
urls: micropurchase-staging.18f.gov
last uploaded: Mon Dec 21 11:57:30 UTC 2015

     state     since                    cpu    memory           disk           details   
#0   running   2015-12-21 06:58:06 AM   0.0%   101.1M of 128M   177.4M of 1G

Good?! Nope. The logs keep showing this same PG::InsufficientPrivilege: ERROR: permission denied to create database error.

Nevertheless, the site was up at the desired URL (https://micropurchase-staging.18f.gov). I tried logging in using our app's OAuth login, and got GitHub's 404 page (at this url: https://github.com/login/oauth/authorize?client_id&redirect_uri=https%3A%2F%2Fmicropurchase-staging.18f.gov%2Fauth%2Fgithub%2Fcallback&response_type=code&state=ec086f1a41afddf44cff5ee885fe8b130034e74c2fe1f00e).

Incidentally cf restage micropurchase gets me:

0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 failing
FAILED
Start unsuccessful

Along with that same PG error.

Why does cf push seem to work fine but cf restage keeps erroring?

Service Name Confusion:

➜  micropurchase git:(master) ✗ cf target

API endpoint:   https://api.cloud.gov (API version: 2.36.0)   
User:           alan.delevie@gsa.gov   
Org:            agile-bpa   
Space:          staging   
➜  micropurchase git:(master) ✗ cf services
Getting services in org agile-bpa / space staging as alan.delevie@gsa.gov...
OK

name                 service   plan          bound apps                                                last operation   
micropurchase-psql   rds       shared-psql   micropurchase-staging, micropurchase-ssh, micropurchase   create succeeded   
➜  micropurchase git:(master) ✗ cf target -s production

API endpoint:   https://api.cloud.gov (API version: 2.36.0)   
User:           alan.delevie@gsa.gov   
Org:            agile-bpa   
Space:          production   
➜  micropurchase git:(master) ✗ cf services
Getting services in org agile-bpa / space production as alan.delevie@gsa.gov...
OK

name                 service       plan          bound apps                                                last operation   
iaa-mongo            mongodb-new   free          iaa-mvp                                                   create succeeded   
micropurchase-psql   rds           shared-psql   micropurchase-staging, micropurchase-ssh, micropurchase   create succeeded

Somehow the same RDS database service is bound to the same three apps (production, staging, and ssh). Maybe this was user (my) error. To correct this, I'll unbind micropurchase-psql from the micropurchase app in the staging space:

➜  micropurchase git:(master) ✗ cf create-service rds shared-psql micropurchase-staging-psql
Creating service micropurchase-staging-psql in org agile-bpa / space staging as alan.delevie@gsa.gov...
OK
➜  micropurchase git:(master) ✗ cf unbind-service micropurchase micropurchase-psql
Unbinding app micropurchase from service micropurchase-psql in org agile-bpa / space staging as alan.delevie@gsa.gov...
OK
➜  micropurchase git:(master) ✗ cf bind-service micropurchase micropurchase-staging-psql
Binding service micropurchase-staging-psql to app micropurchase in org agile-bpa / space staging as alan.delevie@gsa.gov...
OK
TIP: Use 'cf restage' to ensure your env variable changes take effect

Time to restage:

➜  micropurchase git:(master) ✗ cf restage micropurchase
Restaging app micropurchase in org agile-bpa / space staging as alan.delevie@gsa.gov...
[... cf buildpack stuff ...]
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 down
0 of 1 instances running, 1 failing
FAILED
Start unsuccessful

And that same PG error in the logs ...

While I'm at it, let's check cf services to see what's going on:

Getting services in org agile-bpa / space staging as alan.delevie@gsa.gov...
OK

name                         service   plan          bound apps                                 last operation   
micropurchase-psql           rds       shared-psql   micropurchase-staging, micropurchase-ssh   create succeeded   
micropurchase-staging-psql   rds       shared-psql   micropurchase                              create succeeded

Now everything seems backwards: staging db is bound to plain old micropurchase (but in which space?!) and micrpurchase-psql is bound to micropurchase-staging. Again which, space? (cf target says I'm in staging but this is now getting very confusing.

Back out of the rabbit hole

Apologies if this meandered too much, but this seems to be the way things are going (at least for me) with Cloud Foundry and debugging it. So many of the steps required are procedural and not declarative. Maybe this exists already, but I'd much prefer to describe the desired setup in the manfiest.yml: here's what I need in the staging space and here's what I need in production, and then let a space-bound cf push handle the rest.

cloud-gov / cg-site