Launch stage & production environments

[groovecoder]

• [x] Set required environment variables:
  • DATABASE_URL
  • DJANGO_DEBUG=False
  • PUSH_MESSAGES_API_ENDPOINT
  • GOOGLE_ANALYTICS_ACCOUNT
  • RAVEN_DSN
• [ ] Add Firefox Accounts social app
• [x] Run clock process
• [ ] Activate production pontoon

[groovecoder]

@oremj - have you started on any of this yet? This issue may help you in setting up the stage & prod environments.

[oremj]

I just started yesterday afternoon. This is helpful, thanks.

[oremj]

@groovecoder can you please implement the endpoints mentioned in https://github.com/mozilla-services/Dockerflow#containerized-app-requirements, specifically 4 and 5?

The endpoints will need to respond with 200 whether or not X-Forwarded-Proto is set.

I've filed #176 for this.

[oremj]

@groovecoder the set up is pretty close to working. We should enable sentry for tracebacks, also does the app need to send e-mail? If it does, we'll need to add some ENV vars for SMTP configuration.

[groovecoder]

No emails. I'll file an issue for adding sentry.

[oremj]

The site is running, but it's really slow and the workers are timing out. Do you know what could be up?

Apr 13 17:10:52 ip-172-31-55-124 docker[5909]: [2016-04-13 17:10:52 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:98)
Apr 13 17:10:52 ip-172-31-55-124 docker[5909]: [2016-04-13 17:10:52 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:105)
Apr 13 17:10:52 ip-172-31-55-124 docker[5909]: [2016-04-13 17:10:52 +0000] [98] [INFO] Worker exiting (pid: 98)
Apr 13 17:10:52 ip-172-31-55-124 docker[5909]: [2016-04-13 17:10:52 +0000] [105] [INFO] Worker exiting (pid: 105)
Apr 13 17:10:52 ip-172-31-55-124 docker[5909]: [2016-04-13 17:10:52 +0000] [268] [INFO] Booting worker with pid: 268
Apr 13 17:10:52 ip-172-31-55-124 docker[5909]: [2016-04-13 17:10:52 +0000] [269] [INFO] Booting worker with pid: 269

Switching over to debug provides no additional output.

[groovecoder]

Those are the web process workers? Or the clock worker? I do notice that the dev cluster seems slower now too. I'll look into it.

[groovecoder]

@oremj I’m trying to add the new relic agent to the dashboard code so I can analyze why it’s slow. the app is showing up in New Relic (https://rpm.newrelic.com/accounts/263620/applications/13672602) but no data ... do I need to configure AWS/ELB to allow the deis cluster instances to send data out to the new relic IPs & ports?

[groovecoder]

@oremj on the stage server, can you set PUSH_MESSAGES_API_ENDPOINT=http://52.207.213.118? That's the stage stack that @benbangert has going.

[oremj]

The endpoint is configured now.

[groovecoder]

Cool. Is the clock process running? If so, it should automatically retry the start_recording POST to the messages API endpoint every 10 minutes until it succeeds.

Also, is the raven/sentry configured and set up? There should be a messages API connection error from the first time it tried to start_recording POST my app.

[oremj]

I still need to set up the clock process. sentry/raven are configured and I've verified that they are working.

[groovecoder]

Can I get access to the sentry server?

[groovecoder]

@oremj I've also made some updates since you set up the stage server. What's the process workflow for requesting a push/deploy to the stage server?

[oremj]

Right now the process is, ping me. Once we get the log situation, we'll need to start roping QA and we'll use their process.

[groovecoder]

@oremj PING! 😄 can you push the latest code from master to the stage environment? And can I get access to the sentry server?

Also, to whom do I speak in QA about setting up their process for this code & app? @pdehaan ?

[oremj]

master has been pushed to stage.

[groovecoder]

Thanks. @oremj - have you already started a "runbook" type doc on mana?

[groovecoder]

@oremj - where can/should I start a push dev dashboard run-book? I'm being pulled to another project in a couple weeks, so I need to document everything for ongoing ops.

[groovecoder]

@oremj - I wrapped up #209 so I'm ready to help get the prod server up with you and create a run-book for the service.

I think yesterday we decided to run it at https://push-dashboard.services.mozilla.com/ ? Do you need anything else from me to get that up and going?

[groovecoder]

@oremj I added a "run book" section to the docs:

https://push-dev-dashboard.readthedocs.io/en/latest/run.html

Can you scan thru it (and the deploy & testing docs) and see if that covers all the main information you need? If you need anything else in there, comment here, file another issue, and/or submit a PR?