search

aws / opsworks-cookbooks

Chef Cookbooks for the AWS OpsWorks Service
Other
1.05k stars 1.23k forks source link

Opsworks deploy of Rails app with unicorn workers does not erase deleted app environment variables #327

Open ccleung opened 9 years ago

ccleung commented 9 years ago

Steps to reproduce: 1.) Create an opsworks stack for a rails app, which uses unicorn 2.) In the App, set an Environment Variable e.g., name=some_variable 3.) Setup and Deploy an instance 4.) ssh into the deployed instance and verify that the environment variable exists (e.g., log the ENV['some_variable'] in rails app to the log file to see what the value is) 5.) Delete the Environment Variable from Opsworks App page 6.) In Opsworks, Run deploy command (re-deploy the app) 7.) In the instance, observe the shared/config/unicorn.conf, the variable no longer exists, as expected 8.) However, observe ENV['some_variable'] by logging this ENV value to rails log, and looking at the rails logs

Expected: ENV['some_variable'] should be nil

Actual: ENV['some_variable'] is still set

The reason: On a deploy, clean-restart script is run for the unicorn workers: https://github.com/aws/opsworks-cookbooks/blob/release-chef-11.10/deploy/attributes/rails_stack.rb#L28

i.e., https://github.com/aws/opsworks-cookbooks/blob/release-chef-11.10/unicorn/templates/default/unicorn.service.erb#L67

However, if the Gemfile is not changed, only a "soft restart" is issued: https://github.com/aws/opsworks-cookbooks/blob/release-chef-11.10/unicorn/templates/default/unicorn.service.erb#L61

Because of this, unfortunately the unicorn workers will retain ENV variables that are supposed to be erased.

Workaround: Manually executing bash -c '/srv/www/<project_name>/shared/scripts/unicorn stop && /srv/www/<project_name>/shared/scripts/unicorn start'

will update and unset the ENV variables that were deleted from Opsworks App Environment Variables

NOTE: If the Gemfile is updated, it looks like stop and start will be executed for the unicorn workers: https://github.com/aws/opsworks-cookbooks/blob/release-chef-11.10/unicorn/templates/default/unicorn.service.erb#L68

Maybe we should also put in a condition to check for ENV variable change?

Also, it looks like issuing a stop and start on unicorn workers will cause incoming requests to fail temporarily, but this is a separate issue.

ecielam commented 9 years ago

I just ran into this problem with multiple deployments w/o a Gemfile change. The soft restart didn't pick up the new release directory, and when the release it was pointing at was rotated out and deleted we started seeing failures.

stop/start fixed it.