Overriding all tasks - Githubissues

teu commented 8 years ago

Hi,

out deployment cycle relies fully on capistrano, I basically only need shipit to fetch the specific repository. Is there a way to override all shipit events and just run one shell command?

Thanks

byroot commented 8 years ago

Not sure what you mean. Can you be more explicit? What commands shipit currently run, and what do you want it to run?

teu commented 8 years ago

Hi,

just evaluating, having problems as "RAILS_ENV=production rake assets:precompile" doesnt compile assets (they don't show) but in general we have two commands for our deployments:

cap ENV deploy

and

cap ENV rollback VERSION

and this is all I want to run via shipit, no hosts, no installations, nothing, git clone/fetch + cap.

Is this possible? Could not find much in terms of examples.

Cheers

byroot commented 8 years ago

So. A shipit deploy have 3 steps:

Clone / checkout of the repo. There is no way around that one, but you want it anyway so that's ok.
Dependency installation, e.g. install cap. By default shipit will run bundle install if your repo have a Gemfile, but you can customize that step entirely via the yaml file: https://github.com/shopify/shipit-engine#other-dependencies
Then the actual deploy. Shipit will run cap $ENVIRONMENT deploy if it sees a Capfile. But again you can customize it entirely via the YAML: https://github.com/shopify/shipit-engine#deployment

As for rollbacks, we don't infer them so you have to define your own via rollback.override.

So long story short, I believe what you want is pretty much Shipit defaults.

teu commented 8 years ago

Excellent, any idea why my assets don't want to recompile (or rather theserver seems to use some other ones) ?

byroot commented 8 years ago

Well it depends of your capistrano setup. The most common setup is to have capistrano compile the assets on the deployed server.

Are you compiling the assets locally? I'm not sure I understand.

teu commented 8 years ago

The shipit css/js/images used to run fine in development mode - now in production mode, 404's. I ran:

RAILS_ENV=production bundle exec rake assets:precompile

It generated a whole bunch of files but after the server restarted the code still points to 404 statics.

Now I am not a rails dev, perhaps the doc assumes this, and this is a trival lack of rails knowledge.

byroot commented 8 years ago

Oh you are talking about shipit's assets! I though you were talking about the assets of your app.

It generated a whole bunch of files but after the server restarted the code still points to 404 statics.

I see. So it depends how your Shipit instance is deployed, but there is 2 solutions:

If you are behind a web server (Nginx / Apache) then you need to configure your server to serve the content of public in priority (there many articles explaining this around the web).
If you are on Heroku or similar setup where there is no web server on the same machine, then you can set config.serve_static_files = true in config/environments/production.rb. It should already be there with comments around.

Hope it helps.

byroot commented 8 years ago

Here's one article on how to configure nginx for instance: http://vladigleba.com/blog/2014/03/27/deploying-rails-apps-part-4-configuring-nginx/

teu commented 8 years ago

Thanks, a lot, will do that! Perhaps it would be good to mention in the doc so others don't bother with the obvious.

byroot commented 8 years ago

Indeed. We should find a good documentation on how to deploy a Rails app, and reference it from the README.

teu commented 8 years ago

OK, this is done, seems like it's working now!

One more question (sorry to drag it on, perhaps a inFAQ can come out of this).

Added a stack - nothing is happening, got "Refresh scheduled" after clcking "refresh statuses and commits" - any idea, does some outside process need to run for this?

teu commented 8 years ago

OK, figured it out, also good to put sidekiq into the doc. Looks amazing, thanks!

byroot commented 8 years ago

Arf, you beat me to it, I was writing my answer.

And yes I agree the setup docs are totally insuffisant for people not familiar with Rails.

I'll keep this issue open until it's solved.

teu commented 8 years ago

ok, another couple of problems... bear with me here please

started deployment, but dont have a shipit.yml in the repository:

2016-02-17T18:56:46.573Z 15570 TID-tjen0 Shipit::CacheDeploySpecJob JID-a574a29ad85eaf9c524fe996 INFO: start

and thats it - guessing thats due to lack of the config file - any way to stop the process?

byroot commented 8 years ago

Applications can perfectly be deployed even if they don't have a shipit.yml.

CacheDeploySpecJob is triggered for each new commit Shipit sees.

Not sure what your problem is though, is that job stuck? If so it could be because it's trying to clone the repository. You have to make sure the user under which Shipit run can SSH to GitHub and have enough access to clone the repositories.

teu commented 8 years ago

Unsure - the access token has all but team:write permissions:

2016-02-17T19:17:08.969Z 15570 TID-jv9is Shipit::CacheDeploySpecJob JID-7176402b9b4b333747ebf042 INFO: start
2016-02-17T19:22:09.306Z 15570 TID-jv9is Shipit::CacheDeploySpecJob JID-7176402b9b4b333747ebf042 INFO: fail: 300.336 sec
2016-02-17T19:22:09.306Z 15570 TID-jv9is WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"Shipit::CacheDeploySpecJob", "queue"=>"default", "args"=>[{"job_class"=>"Shipit::CacheDeploySpecJob", "job_id"=>"e0463281-3313-41af-adb8-0815a72df255", "queue_name"=>"default", "arguments"=>[{"_aj_globalid"=>"gid://shipit/Shipit::Stack/1"}], "locale"=>"en"}], "retry"=>true, "jid"=>"7176402b9b4b333747ebf042", "created_at"=>1455735723.0405533, "enqueued_at"=>1455735723.0406227, "error_message"=>"Input/output error @ io_getpartial - /dev/pts/13", "error_class"=>"Errno::EIO", "failed_at"=>1455736929.305984, "retry_count"=>0}
2016-02-17T19:22:09.306Z 15570 TID-jv9is WARN: Errno::EIO: Input/output error @ io_getpartial - /dev/pts/13
2016-02-17T19:22:09.307Z 15570 TID-jv9is WARN: /var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:117:in `readpartial'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:117:in `block (2 levels) in read_stream'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `block in catch'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `catch'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `catch'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:103:in `timeout'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:126:in `with_timeout'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:116:in `block in read_stream'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:115:in `loop'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:115:in `read_stream'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:147:in `block in kill_and_wait'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `block in catch'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `catch'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `catch'
/usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:103:in `timeout'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:146:in `kill_and_wait'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:130:in `terminate!'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:89:in `rescue in stream'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:84:in `stream'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:109:in `stream!'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:52:in `run!'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/stack_commands.rb:37:in `block in with_temporary_working_directory'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/redis-objects-1.2.1/lib/redis/lock.rb:58:in `lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/models/shipit/stack.rb:173:in `acquire_git_cache_lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/stack_commands.rb:36:in `with_temporary_working_directory'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/cache_deploy_spec_job.rb:7:in `perform'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:32:in `block in perform_now'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/redis-objects-1.2.1/lib/redis/lock.rb:58:in `lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/background_job/unique.rb:20:in `acquire_lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/background_job/unique.rb:9:in `block (2 levels) in <module:Unique>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `block in tag_logger'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:26:in `tagged'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `tagged'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `tag_logger'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_perform_callbacks'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:31:in `perform_now'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:21:in `execute'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:150:in `execute_job'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/logging.rb:30:in `with_context'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:132:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:132:in `invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:127:in `block in process'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:166:in `stats'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:126:in `process'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:79:in `process_one'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:67:in `run'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/util.rb:16:in `watchdog'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'

teu commented 8 years ago

Or do you mean to clone it via ssh?

byroot commented 8 years ago

Yes I confirm, it definitely hanged during cloning.

Try to SSH on Shipit's server and then:

su <whtever-user-shipit-runs-under>
ssh -vvvv git@github.com

You can also try a git clone.

byroot commented 8 years ago

Or do you mean to clone it via ssh?

Yes, I though the docs mentioned it, but turns out it doesn't. The unix user that run Shipit needs to have enough SSH credentials to be able to git clone git@github.com:your-repo.

teu commented 8 years ago

At this point it will not surprise anyone when I say I have another problem:

(...)

HEAD is now at e3c3abc... same name of stage and file

$ sed -i '/^ruby\s/d' Gemfile
pid: 17742

$ bundle check --path=/var/capistrano/shipit/data/bundler || bundle install --frozen --path=/var/capistrano/shipit/data/bundler --retry=2 --without=default:production:development:test:staging:benchmark:debug
pid: 17745
git@github.com:teu/capistrano-asgroup.git (at master@ef48faf) is not yet checked
out. Run `bundle install` first.
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Using bundler 1.11.2
Bundle complete! 7 Gemfile dependencies, 1 gem now installed.
Gems in the groups default, production, development, test, staging, benchmark and debug were not installed.
Bundled gems are installed into /var/capistrano/shipit/data/bundler.

$ bundle exec cap $ENVIRONMENT deploy
pid: 17765
bundler: command not found: cap
Install missing gem executables with `bundle install`
bundle exec cap $ENVIRONMENT deploy exited with status 127

byroot commented 8 years ago

At this point it will not surprise anyone when I say I have another problem:

Lol, no problem, I'm happy to help.

Ok, so this is a relatively common one. It's explained here: https://github.com/shopify/shipit-engine#installing-dependencies

By default the following gem groups will be ignored: ... The gems you need in order to deploy should be in a different group, such as deploy.

So you need to put capistrano and other gems required for deploy in the deploy group.

teu commented 8 years ago

One step closer ;-) It got to fire up capistrano. Our deployment does some internal checkouts to temp dirs (assuming this can change as shipit already has a temp dir) and performs some additional clones and so on. The root user which it runs on also has access to the repos.

2016-02-17T20:39:38.901Z 17793 TID-ov7sj9sy0 Shipit::CacheDeploySpecJob JID-6d7b5d37d3ffdbbd20be5be9 INFO: start
2016-02-17T20:39:41.394Z 17793 TID-ov7sj9sy0 Shipit::CacheDeploySpecJob JID-6d7b5d37d3ffdbbd20be5be9 INFO: fail: 2.494 sec
2016-02-17T20:39:41.395Z 17793 TID-ov7sj9sy0 WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"Shipit::CacheDeploySpecJob", "queue"=>"default", "args"=>[{"job_class"=>"Shipit::CacheDeploySpecJob", "job_id"=>"0a793a5d-f444-4589-b8d0-39539241e531", "queue_name"=>"default", "arguments"=>[{"_aj_globalid"=>"gid://shipit/Shipit::Stack/1"}], "locale"=>"en"}], "retry"=>true, "jid"=>"6d7b5d37d3ffdbbd20be5be9", "created_at"=>1455740831.3200357, "enqueued_at"=>1455741578.8999, "error_message"=>"git checkout --force origin/asg_deployment_test exited with status 1", "error_class"=>"Shipit::Command::Error", "failed_at"=>1455740834.0385802, "retry_count"=>5, "retried_at"=>1455741581.3925164}
2016-02-17T20:39:41.395Z 17793 TID-ov7sj9sy0 WARN: Shipit::Command::Error: git checkout --force origin/asg_deployment_test exited with status 1
2016-02-17T20:39:41.395Z 17793 TID-ov7sj9sy0 WARN: /var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:110:in `stream!'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/command.rb:52:in `run!'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/stack_commands.rb:38:in `block in with_temporary_working_directory'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/redis-objects-1.2.1/lib/redis/lock.rb:58:in `lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/models/shipit/stack.rb:173:in `acquire_git_cache_lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/lib/shipit/stack_commands.rb:36:in `with_temporary_working_directory'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/cache_deploy_spec_job.rb:7:in `perform'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:32:in `block in perform_now'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/redis-objects-1.2.1/lib/redis/lock.rb:58:in `lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/background_job/unique.rb:20:in `acquire_lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/background_job/unique.rb:9:in `block (2 levels) in <module:Unique>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `block in tag_logger'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:26:in `tagged'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `tagged'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `tag_logger'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_perform_callbacks'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:31:in `perform_now'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:21:in `execute'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activejob-4.2.5.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:150:in `execute_job'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/logging.rb:30:in `with_context'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:132:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/middleware/chain.rb:132:in `invoke'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:127:in `block in process'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:166:in `stats'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:126:in `process'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:79:in `process_one'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/processor.rb:67:in `run'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/util.rb:16:in `watchdog'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/sidekiq-4.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'

and blocking again - fails at a few retries.

byroot commented 8 years ago

So clearly it failed to checkout the head of the branch you asked to track:

git checkout --force origin/asg_deployment_test exited with status 1

Does that branch exist? The best way to debug this is probably to clone that repo with the same user shipit use and then try to run git checkout --force origin/asg_deployment_test.

teu commented 8 years ago

This is the sam branch that ha been checked out by shipit - basically our cap made sure it doesnt work on the local checkout as this could be on someones machine. Shipit runs as root.

teu commented 8 years ago

I can confirm I can check out the repository on the same user and fetch the branch just fine:

git clone...
cd [checkout]
git checkout --force origin/asg_deployment_test
echo $? // result 0

teu commented 8 years ago

The weird thing is I am not doing that in capistrano, the first thing I am doing is git clone to a temp folder and never force. Capistrano was clearly an by shipit (as I got output about the servers it found to deploy to) so I am not sure about the precedence of events here.

teu commented 8 years ago

Stuck on this - any idea?

byroot commented 8 years ago

The weird thing is I am not doing that in capistrano, the first thing I am doing is git clone to a temp folder and never force.

Shipit does that to read the YAML file. That is what CacheDeploySpecJob is for.

Stuck on this - any idea?

Well, I'm not entirely sure to understand where you're at. You triggered a deploy and it crashed, right?

If so can you paste the entire deploy output in a gist? It will give me a better idea.

teu commented 8 years ago

Hi,

any change of sending this via email so it's not public?

byroot commented 8 years ago

Sure. Feel free to contact me on my public GitHub email.

teu commented 8 years ago

On the other hand this does not to appear to have anything sensitive. https://gist.github.com/teu/4fba4e43de01937b5db9

byroot commented 8 years ago

Ok, so according to that gist shipit attempted to kill capistrno because it didn't print anything in 5 minutes. Look like capistrano is hanging on something

teu commented 8 years ago

Yes, looking at sidekiq.log:

2016-02-17T22:21:33.031Z 15570 TID-ouuue0j44 Shipit::CacheDeploySpecJob JID-fe401d73b9a8967ac349dc52 INFO: start
2016-02-17T22:21:33.039Z 17793 TID-ov7u6iioc Shipit::CacheDeploySpecJob JID-7176402b9b4b333747ebf042 INFO: start
2016-02-17T22:21:33.046Z 17793 TID-ov7u6iioc Shipit::CacheDeploySpecJob JID-7176402b9b4b333747ebf042 INFO: fail: 0.007 sec
2016-02-17T22:21:33.046Z 17793 TID-ov7u6iioc WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"Shipit::CacheDeploySpecJob", "queue"=>"default", "args"=>[{"job_class"=>"Shipit::CacheDeploySpecJob", "job_id"=>"e0463281-3313-41af-adb8-0815a72df255", "queue_name"=>"default", "arguments"=>[{"_aj_globalid"=>"gid://shipit/Shipit::Stack/1"}], "locale"=>"en"}], "retry"=>true, "jid"=>"7176402b9b4b333747ebf042", "created_at"=>1455735723.0405533, "enqueued_at"=>1455747693.0364296, "error_message"=>"Timeout on lock Shipit::CacheDeploySpecJob-{\"_aj_globalid\"=>\"gid://shipit/Shipit::Stack/1\"} exceeded 0 sec", "error_class"=>"Redis::Lock::LockTimeout", "failed_at"=>1455736929.305984, "retry_count"=>9, "retried_at"=>1455747693.0463693}
2016-02-17T22:21:33.046Z 17793 TID-ov7u6iioc WARN: Redis::Lock::LockTimeout: Timeout on lock Shipit::CacheDeploySpecJob-{"_aj_globalid"=>"gid://shipit/Shipit::Stack/1"} exceeded 0 sec
2016-02-17T22:21:33.047Z 17793 TID-ov7u6iioc WARN: /var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/redis-objects-1.2.1/lib/redis/lock.rb:87:in `try_until_timeout'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/redis-objects-1.2.1/lib/redis/lock.rb:34:in `lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/background_job/unique.rb:20:in `acquire_lock'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/jobs/shipit/background_job/unique.rb:9:in `block (2 levels) in <module:Unique>'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/var/capistrano/shipit/vendor/bundle/ruby/2.2.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'

byroot commented 8 years ago

That's a different problem I'm afraid. Would you have time for a call a bit later? It would probably be easier / faster

teu commented 8 years ago

Hi, sure, will email you with my skype.

byroot commented 8 years ago

So for pulbic record, https://github.com/mattbrictson/airbrussh was causing issues when invoked from shipit. I'll file a specific issue for this.

teu commented 8 years ago

After some tinkering got it a lot further. Managed to quickly get the deployment to work from the local checkout (probably good to mention the initial steps shipit is taking with the code in the doc) it failed on yet another thing that works fine from the same user:

INFO [c670e7c7] Running /usr/bin/env cd . && php composer.phar --prefer-dist --no-dev --no-interaction --optimize-autoloader install as teu via Shipit@localhost
DEBUG [c670e7c7] Command: cd . && php composer.phar --prefer-dist --no-dev --no-interaction --optimize-autoloader install
DEBUG [c670e7c7]    sh: 1: php: not found
cap aborted!
SSHKit::Command::Failed: cd . && php composer.phar --prefer-dist --no-dev --no-interaction --optimize-autoloader install exit status: 32512

Can I ask for help on this one?

byroot commented 8 years ago

The error is here: sh: 1: php: not found

Since it works when you run it manually, I think it's because shipit doesn't spawn commands in a shell and as such isn't sourcing /etc/environment, and other ~/.bash_profile. That problem is very common in cron tasks as well.

To confirm that, you can make shipit print environment variables:

deploy:
  override:
    - echo $PATH
    - bundle exec cap $ENVIRONMENT deploy

byroot commented 8 years ago

Oh and if I'm right, then you need to make sure Shipit's $PATH is set properly. There is many way to do it though all depending on how you shipit instance is hosted.

But a shitty hacky way could be:

# in config/initializers/env.rb

ENV['PATH'] = "#{ENV['PATH']}:/path/to/php/install/dir"

teu commented 8 years ago

You are correct, missed it in phpfarm settings. Many thanks!

Found this for capistrano

set :default_environment, {
  'PATH' => "/opt/ruby-enterprise/bin/:$PATH"
}

teu commented 8 years ago

Hmm, the path in /etc/environment is set for a long time, not sure why it doesn't work. The set in capistrano also did not work, looks like something is overriding it.

byroot commented 8 years ago

Did you set "/opt/ruby-enterprise/bin/:$PATH" exactly or is it an example?

teu commented 8 years ago

No:) It's

set :default_environment, {
  'PATH' => "$PATH:/opt/phpfarm/inst/current-bin"
}

byroot commented 8 years ago

A bunch of ideas here.

If you are using capistrano 3, then I think the variable is default_env, not default_environment
From the logs you pasted I believe php is invoked locally. I'm not 100% sure capistrano use that variable for local commands. You can also try ENV['PATH'] = "#{ENV['PATH']}:/path/to/php/install/dir" inside you cap recipe.

teu commented 8 years ago

Weird thing is that /etc/environment is overriden, shouldn't it be the base for /usr/bin/env?

byroot commented 8 years ago

I'm not familiar enough with usr/bin/env but I don't think it sources /etc/environment.

teu commented 8 years ago

Think its some rvm magic.

teu commented 8 years ago

This got me further, thanks. Two more issues:

at some point the cli stream gets stuck (~3-4 minutes into the process) Looked into it and
some of the responses of the xhr start responding with 304, not sure if that is indended in case of nothing new to report or is there a cache problem
after some point 500 are being returned:

'1678')  ORDER BY `output_chunks`.`id` ASC  [["task_id", 21]]
I, [2016-02-18T22:12:06.085848 #20429]  INFO -- : Completed 500 Internal Server Error in 26ms (ActiveRecord: 10.7ms)
F, [2016-02-18T22:12:06.088510 #20429] FATAL -- :
Encoding::UndefinedConversionError ("\xE2" from ASCII-8BIT to UTF-8):
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/core_ext/object/json.rb:34:in `encode'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/core_ext/object/json.rb:34:in `to_json'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/core_ext/object/json.rb:34:in `to_json_with_active_support_encoder'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/json/encoding.rb:57:in `to_json'
  vendor/bundle/ruby/2.2.0/gems/json-1.8.3/lib/json/common.rb:223:in `generate'
  vendor/bundle/ruby/2.2.0/gems/json-1.8.3/lib/json/common.rb:223:in `generate'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/json/encoding.rb:101:in `stringify'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/json/encoding.rb:35:in `encode'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/json/encoding.rb:22:in `encode'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/core_ext/object/json.rb:37:in `to_json_with_active_support_encoder'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/renderers.rb:116:in `block in <module:Renderers>'
  vendor/bundle/ruby/2.2.0/gems/active_model_serializers-0.9.4/lib/action_controller/serialization.rb:55:in `block (2 levels) in <module:Serialization>'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/renderers.rb:45:in `block in _render_to_body_with_renderer'
  /usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/set.rb:283:in `each_key'
  /usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/set.rb:283:in `each'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/renderers.rb:41:in `_render_to_body_with_renderer'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/renderers.rb:37:in `render_to_body'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/abstract_controller/rendering.rb:25:in `render'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rendering.rb:16:in `render'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/core_ext/benchmark.rb:12:in `block in ms'
  /usr/local/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/benchmark.rb:303:in `realtime'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/core_ext/benchmark.rb:12:in `ms'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
  vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.5.1/lib/active_record/railties/controller_runtime.rb:25:in `cleanup_view_runtime'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:43:in `render'
  vendor/bundle/ruby/2.2.0/bundler/gems/shipit-engine-8b51fed7f4e9/app/controllers/shipit/tasks_controller.rb:41:in `tail'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:198:in `process_action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rendering.rb:10:in `process_action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rescue.rb:29:in `process_action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
  vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.5.1/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in `process'
  vendor/bundle/ruby/2.2.0/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in `process'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:196:in `dispatch'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:237:in `block in action'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:43:in `serve'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:43:in `block in serve'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `each'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `serve'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:815:in `call'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/engine.rb:518:in `call'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `public_send'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `method_missing'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/mapper.rb:51:in `serve'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:43:in `block in serve'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `each'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `serve'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:815:in `call'
  vendor/bundle/ruby/2.2.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
  vendor/bundle/ruby/2.2.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
  vendor/bundle/ruby/2.2.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/etag.rb:24:in `call'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:25:in `call'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/flash.rb:260:in `call'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/cookies.rb:560:in `call'
  vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.5.1/lib/active_record/query_cache.rb:36:in `call'
  vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:38:in `call_app'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:20:in `block in call'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:26:in `tagged'
  vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `tagged'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:20:in `call'
  vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/request_id.rb:21:in `call'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
  vendor/bundle/ruby/2.2.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/engine.rb:518:in `call'
  vendor/bundle/ruby/2.2.0/gems/railties-4.2.5.1/lib/rails/application.rb:165:in `call'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

Using mysql with an utf8 DB may be part of the problem (also probably good to add taht gem/setup doc).

abort doesn't abort, said it sends sigterm but does not stop the process

On the other hand the shipit ship took off and I finally got a deploy through, hurray! :)

byroot commented 8 years ago

Encoding::UndefinedConversionError ("\xE2" from ASCII-8BIT to UTF-8):

Interesting. The encoding of the commands output is off. I'll investigate a bit more about this, but at first sight it is likely related to $LANG / $LC_ALL. Can you add:

p ENV['LANG']
p ENV['LC_ALL']

In your capistrano, please?

teu commented 8 years ago

Those values probably need to be set system wide (had a tool with the same problem just now).

Shopify / shipit-engine

Overriding all tasks #527