search

capistrano / rvm

MIT License
140 stars 47 forks source link

Any remote RVM commands generate hundreds of wrapper RVM instances and recurse forever #27

Closed coneybeare closed 10 years ago

coneybeare commented 10 years ago

I am running a simple cap production rvm:check.

Locally, everything looks good until it hits here:

[a0763390] Running /usr/local/rvm/bin/rvm ruby-2.0.0-p353@[REDACTED] do ruby --version on [REDACTED].com
DEBUG [a0763390] Command: ( RAILS_ENV=production /usr/local/rvm/bin/rvm ruby-2.0.0-p353@[REDACTED] do ruby --version )

ruby-2.0.0-p353 is installed on the remote machine, as is the gemset.

On the server side, hundreds of rvm wrapper instances are spawned per second with no end. Only a force kill locally, and a pkill remotely will cause it to stop.

$ root@[REDACTED]:/home/[REDACTED]/[REDACTED].com/current# ps ax | grep rvm
  546 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
  684 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
  791 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
  898 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1005 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1112 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1220 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1341 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1452 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1559 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1666 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1779 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1886 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 1993 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2101 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2208 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2315 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2437 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2544 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2651 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2760 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2869 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 2980 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3087 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3196 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3308 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3416 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3530 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3646 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3754 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3861 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 3969 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4078 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4187 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4294 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4402 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4509 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4617 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4734 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4841 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 4970 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5078 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5185 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5292 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5401 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5508 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5615 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5722 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5829 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 5936 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 6043 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
 6050 pts/2    S+     0:00 grep rvm
25484 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
25850 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
25957 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26064 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26171 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26279 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26386 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26493 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26600 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26707 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26814 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
26921 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27028 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27135 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27242 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27349 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27456 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27563 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27670 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27777 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27884 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
27992 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28100 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28208 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28315 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28422 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28529 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28636 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28743 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28851 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
28959 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29066 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29173 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29280 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29387 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29494 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29601 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29708 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29815 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
29922 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30029 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30136 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30243 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30350 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30457 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30565 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30672 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30779 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30887 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
30994 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31101 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31208 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31315 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31422 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31529 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31636 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31743 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31850 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
31957 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
32064 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
32171 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
32278 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
32386 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
32493 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
32600 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
32707 ?        S      0:00 bash /usr/local/rvm/scripts/wrapper ruby-2.0.0-p353@[REDACTED]
$ root@[REDACTED]:/home/[REDACTED]/[REDACTED].com/current# ps ax | grep rvm | awk '{ print $1 }' | xargs kill

I never had a problem with cap 2. What is going on here?

mpapis commented 10 years ago

did you tried running just the command on server:

/usr/local/rvm/bin/rvm ruby-2.0.0-p353@[REDACTED] do ruby --version
coneybeare commented 10 years ago

I just ran it and only 2 wrapper instances were spawned, not the same behavior as when running through cap

coneybeare commented 10 years ago

and in case it wasn't clear, the version string was printed to console

mpapis commented 10 years ago

what is surprising is that it's called at all, check if you have any executable hooks in rvm:

ls -l $rvm_path/hooks

those with executable bit are enabled (x), most of the hooks is disabled by default, but some users report that running chmod -R +x $rvm_path does it .. the easiest is to disable all the hooks with: chmod -x $rvm_path/hooks/*

coneybeare commented 10 years ago

They are all default installed hooks. The executables are after_install_auto_gem, after_install_codesign and after_use_maglev.

I have been tinkering around with the server side and reinstalled rvm, reinstalled ruby-2.0.0-p353, manually redid a bundle install and now the error is gone. The bug, if exists, could still be there, but I can no longer reproduce it. If the source is not able to be determined, perhaps a local check for the symptoms could be added. I only noticed because it slowed down every other site on my server, then I realized there were tens of thousands of rvm wrapper instances happening. It seems like it could have been prevented with a default timeout on the local side.

mpapis commented 10 years ago

maybe this was older version of rvm?

coneybeare commented 10 years ago

I don't think that is the case, I ran a rvm get head yesterday

mpapis commented 10 years ago

if anyone can reproduce it then please report a ticket for rvm => https://github.com/wayneeseguin/rvm/issues

kirs commented 10 years ago

@coneybeare I would recommend you switching to rbenv which is super lightweight and don't have such many issues like overriding cd command.

mpapis commented 10 years ago

@kirs since when overriding cd is a problem? it is totally unrelated to this problem, and the rvm override of cd works perfectly fine, you could find a lot of issues in rvm but you had to mention not-an-issue.

coneybeare commented 10 years ago

I am generally pretty comfortable running RVM so I will stick with it, but thanks

mpapis commented 10 years ago

@coneybeare there is alternative (better) implementation of the integration that should work better https://github.com/rvm/rvm1-capistrano3 ... I'm missing time to open a PR to merge it over here.