search

thoughtbot / cocaine

A small library for doing (command) lines.
https://robots.thoughtbot.com
Other
785 stars 55 forks source link

Multi threading support? #23

Closed geoffw8 closed 12 years ago

geoffw8 commented 12 years ago

Hey there,

I'm a heavy user of Sidekiq and Paperclip. We get an absolute ton of errors come through like this:

Errno::EINVAL: Invalid argument - identify -format %wx%h '/tmp/stream20120911-6678-1uwdq83.txt[0]'

I'm talking thousands. While in theory I could be running 1000 workers (which would be ideal as I need to process just over 1m images) I find that things run slightly smoother if I run just 50.

Disclaimer: Now I'm a total amateur, really my knowledge stops dead at Ruby/Rails, I know my way around a shell but I wont be winning any awards any time soon.

I never used to get this error with Resque and Paperclip, and so I'm wondering if its a volume thing (back then we only ran 20 workers) and they all worked very well, this was never an issue. Is there something at play here? I wonder if cocaine is struggling to speak to ImageMagick quick enough, (or something, I know thats not too technical!).

I don't know if you recall but I actually asked this question before and you pointed me at a certain commit:

gem "cocaine", :git => "http://github.com/thoughtbot/cocaine.git", :ref => "26a64a9d67a6fcac76349c13167dcddb6596f702"

Would appreciate any pointers... is where my gemfile pointed still valid? I recall you were pondering bumping the version.

Thanks in advance, getting these images processing would be amazing!

Regards,

Geoff

jyurek commented 12 years ago

Hello! First off, there have been more commits to cocaine since then, so you can update to either version 0.3.0 of the gem or to the commit currently at master. Where you have it is still "valid", it's just not the latest.

Second, cocaine wouldn't be having a problem with volume. "Invalid argument" isn't an error it would give if it can't keep up. I'm not sure what is actually giving that error, so it would be very helpful if you could paste a stack trace.

On Sep 11, 2012, at 10:07 AM, geoffw8 wrote:

Hey there,

I'm a heavy user of Sidekiq and Paperclip. We get an absolute ton of errors come through like this:

Errno::EINVAL: Invalid argument - identify -format %wx%h '/tmp/stream20120911-6678-1uwdq83.txt[0]'

I'm talking thousands. While in theory I could be running 1000 workers (which would be ideal as I need to process just over 1m images) I find that things run slightly smoother if I run just 50.

Disclaimer: Now I'm a total amateur, really my knowledge stops dead at Ruby/Rails, I know my way around a shell but I wont be winning any awards any time soon.

I never used to get this error with Resque and Paperclip, and so I'm wondering if its a volume thing (back then we only ran 20 workers) and they all worked very well, this was never an issue. Is there something at play here? I wonder if cocaine is struggling to speak to ImageMagick quick enough, (or something, I know thats not too technical!).

I don't know if you recall but I actually asked this question before and you pointed me at a certain commit:

gem "cocaine", :git => "http://github.com/thoughtbot/cocaine.git", :ref => "26a64a9d67a6fcac76349c13167dcddb6596f702"

Would appreciate any pointers... is where my gemfile pointed still valid? I recall you were pondering bumping the version.

Thanks in advance, getting these images processing would be amazing!

Regards,

Geoff

— Reply to this email directly or view it on GitHub.

Jonathan Yurek, Co-founder thoughtbot, inc. 617.482.1300 x114 http://thoughtbot.com/ http://twitter.com/thoughtbot

geoffw8 commented 12 years ago

Hi Jon,

Thanks for getting back to me :)

Heres the stack trace:

[GEM_ROOT]/bundler/gems/cocaine-26a64a9d67a6/lib/cocaine/command_line/runners/process_runner.rb:17

[GEM_ROOT]/bundler/gems/cocaine-26a64a9d67a6/lib/cocaine/command_line/runners/process_runner.rb:17

[GEM_ROOT]/bundler/gems/cocaine-26a64a9d67a6/lib/cocaine/command_line/runners/process_runner.rb:8

[GEM_ROOT]/bundler/gems/cocaine-26a64a9d67a6/lib/cocaine/command_line.rb:77

[GEM_ROOT]/bundler/gems/cocaine-26a64a9d67a6/lib/cocaine/command_line.rb:55

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip.rb:102

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/geometry.rb:22

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/thumbnail.rb:35

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/processor.rb:33

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/processor.rb:33

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:441

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:440

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:440

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:440

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:433

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:432

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:432

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:425

[GEM_ROOT]/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:390

[GEM_ROOT]/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:81

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/callback_compatibility.rb:54

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:424

[GEM_ROOT]/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:399

[GEM_ROOT]/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:81

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/callback_compatibility.rb:54

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:423

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:120

[GEM_ROOT]/gems/paperclip-2.7.0/lib/paperclip.rb:349

[PROJECT_ROOT]/app/models/product.rb:605

[PROJECT_ROOT]/app/workers/get_photo.rb:10

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/processor.rb:41

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:79

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:79

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/server/timeout.rb:14

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:81

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/server/active_record.rb:6

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:81

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/server/retry_jobs.rb:42

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:81

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/server/logging.rb:11

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/logging.rb:22

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/server/logging.rb:7

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:81

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:84

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/middleware/chain.rb:84

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/processor.rb:40

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/processor.rb:74

[GEM_ROOT]/gems/sidekiq-2.3.0/lib/sidekiq/processor.rb:39

[GEM_ROOT]/gems/celluloid-0.12.0/lib/celluloid/calls.rb:99

[GEM_ROOT]/gems/celluloid-0.12.0/lib/celluloid/actor.rb:320

[GEM_ROOT]/gems/celluloid-0.12.0/lib/celluloid/tasks/task_fiber.rb:22

And OK, I'll update my version now.

I'm guessing the reason cocaine is top of the stack trace is because its cocaine itself that receives the error from the OS right?

Thanks, Geoff

On Tue, Sep 11, 2012 at 3:42 PM, Jon Yurek notifications@github.com wrote:

Hello! First off, there have been more commits to cocaine since then, so you can update to either version 0.3.0 of the gem or to the commit currently at master. Where you have it is still "valid", it's just not the latest.

Second, cocaine wouldn't be having a problem with volume. "Invalid argument" isn't an error it would give if it can't keep up. I'm not sure what is actually giving that error, so it would be very helpful if you could paste a stack trace.

On Sep 11, 2012, at 10:07 AM, geoffw8 wrote:

Hey there,

I'm a heavy user of Sidekiq and Paperclip. We get an absolute ton of errors come through like this:

Errno::EINVAL: Invalid argument - identify -format %wx%h '/tmp/stream20120911-6678-1uwdq83.txt[0]'

I'm talking thousands. While in theory I could be running 1000 workers (which would be ideal as I need to process just over 1m images) I find that things run slightly smoother if I run just 50.

Disclaimer: Now I'm a total amateur, really my knowledge stops dead at Ruby/Rails, I know my way around a shell but I wont be winning any awards any time soon.

I never used to get this error with Resque and Paperclip, and so I'm wondering if its a volume thing (back then we only ran 20 workers) and they all worked very well, this was never an issue. Is there something at play here? I wonder if cocaine is struggling to speak to ImageMagick quick enough, (or something, I know thats not too technical!).

I don't know if you recall but I actually asked this question before and you pointed me at a certain commit:

gem "cocaine", :git => "http://github.com/thoughtbot/cocaine.git", :ref => "26a64a9d67a6fcac76349c13167dcddb6596f702"

Would appreciate any pointers... is where my gemfile pointed still valid? I recall you were pondering bumping the version.

Thanks in advance, getting these images processing would be amazing!

Regards,

Geoff

— Reply to this email directly or view it on GitHub.

Jonathan Yurek, Co-founder thoughtbot, inc. 617.482.1300 x114 http://thoughtbot.com/ http://twitter.com/thoughtbot

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8462272.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

jyurek commented 12 years ago

Ok, after a bit of digging, I may have found something. Can you try to use this branch in your Gemfile?

gem 'cocaine', :git => "http://github.com/thoughtbot/cocaine.git", :branch => "23-maybe-fix"

And see if the error pops up?

jyurek commented 12 years ago

Hi, have you tried this new branch or the latest version of Paperclip to see if ti works?

geoffw8 commented 12 years ago

Hi Jon,

Sorry, I've been away. I'll give it a go right now.

Thanks,

Geoff

On Fri, Sep 14, 2012 at 3:47 PM, Jon Yurek notifications@github.com wrote:

Hi, have you tried this new branch or the latest version of Paperclip to see if ti works?

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8563438.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

geoffw8 commented 12 years ago

Hi Jon,

I deployed this yesterday, and have done some image processing this morning, and at first glance it doesn't appear to have worked. In fact, I seem to have a worse error than before, only this time it looks like a real hard-core error (excuse the non-techy speak) I got heap, a list of all the files in the project and a "Report this to Ruby-lang" type message - so I guess its a Ruby error.

I ran 4k images, 2k worked perfectly, I was checking in mySQL as well as our Sidekiq web panel and all was going well, then, all of a sudden they start failing (every single one), and now I have 2k failed, and 2k working. Same code, nothings changed. The images were from the same store (Nike!) so the formats/availability etc of them is certain.

I'm a bit muddled, I notice this is a thoughtbot gem and so a cousin of Paperclip and I must say I'm not feeling too confident about Paperclip at the moment! My post over there has gone unanswered. I appreciate people are busy... but I dont seem to be the only person having this problem!

I appreciate we might be an edge case (?), we're literally running tens of thousands of these a day, so I thought I'd scale it back to just a few hundred, or a few thousand.

What do you think might be the problem? Do you think it might be a cocaine thing? I guess I don't really know where to start!

Kind regards,

(and thanks for your help, it really is much appreciated)

Geoff

On Tue, Sep 18, 2012 at 6:16 PM, Geoff Wright geoff@weartolook.com wrote:

Hi Jon,

Sorry, I've been away. I'll give it a go right now.

Thanks,

Geoff

On Fri, Sep 14, 2012 at 3:47 PM, Jon Yurek notifications@github.comwrote:

Hi, have you tried this new branch or the latest version of Paperclip to see if ti works?

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8563438.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

geoffw8 commented 12 years ago

Heh, I just noticed, you were the last guy to commit to Paperclip...!

On Wed, Sep 19, 2012 at 4:03 PM, Geoff Wright geoff@weartolook.com wrote:

Hi Jon,

I deployed this yesterday, and have done some image processing this morning, and at first glance it doesn't appear to have worked. In fact, I seem to have a worse error than before, only this time it looks like a real hard-core error (excuse the non-techy speak) I got heap, a list of all the files in the project and a "Report this to Ruby-lang" type message - so I guess its a Ruby error.

I ran 4k images, 2k worked perfectly, I was checking in mySQL as well as our Sidekiq web panel and all was going well, then, all of a sudden they start failing (every single one), and now I have 2k failed, and 2k working. Same code, nothings changed. The images were from the same store (Nike!) so the formats/availability etc of them is certain.

I'm a bit muddled, I notice this is a thoughtbot gem and so a cousin of Paperclip and I must say I'm not feeling too confident about Paperclip at the moment! My post over there has gone unanswered. I appreciate people are busy... but I dont seem to be the only person having this problem!

I appreciate we might be an edge case (?), we're literally running tens of thousands of these a day, so I thought I'd scale it back to just a few hundred, or a few thousand.

What do you think might be the problem? Do you think it might be a cocaine thing? I guess I don't really know where to start!

Kind regards,

(and thanks for your help, it really is much appreciated)

Geoff

On Tue, Sep 18, 2012 at 6:16 PM, Geoff Wright geoff@weartolook.comwrote:

Hi Jon,

Sorry, I've been away. I'll give it a go right now.

Thanks,

Geoff

On Fri, Sep 14, 2012 at 3:47 PM, Jon Yurek notifications@github.comwrote:

Hi, have you tried this new branch or the latest version of Paperclip to see if ti works?

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8563438.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

jyurek commented 12 years ago

Yes, Paperclip is also "my" project. I apologize for not getting to your post there.

Anyway, from the looks of it, you're not having the same problem they are. That thread is about Errno::E2BIG errors and you're getting EINVAL. Now, neither make sense from what I know of the commands that are being fed to the shells, so in that way they're similar, but the errors being reported differ.

I did make a change in another commit, so if you mind updating your gem to cocaine/master and putting "Cocaine::CommandLine.runner = Cocaine::CommandLine::BackticksRunner.new" into config/initializers. This will return the functionality to pre-0.3.0 and should allow you to run everything you need to, at least.

Why Process.spawn is such a pain, I still can't say.

On Sep 19, 2012, at 11:12 AM, geoffw8 wrote:

Heh, I just noticed, you were the last guy to commit to Paperclip...!

On Wed, Sep 19, 2012 at 4:03 PM, Geoff Wright geoff@weartolook.com wrote:

Hi Jon,

I deployed this yesterday, and have done some image processing this morning, and at first glance it doesn't appear to have worked. In fact, I seem to have a worse error than before, only this time it looks like a real hard-core error (excuse the non-techy speak) I got heap, a list of all the files in the project and a "Report this to Ruby-lang" type message - so I guess its a Ruby error.

I ran 4k images, 2k worked perfectly, I was checking in mySQL as well as our Sidekiq web panel and all was going well, then, all of a sudden they start failing (every single one), and now I have 2k failed, and 2k working. Same code, nothings changed. The images were from the same store (Nike!) so the formats/availability etc of them is certain.

I'm a bit muddled, I notice this is a thoughtbot gem and so a cousin of Paperclip and I must say I'm not feeling too confident about Paperclip at the moment! My post over there has gone unanswered. I appreciate people are busy... but I dont seem to be the only person having this problem!

I appreciate we might be an edge case (?), we're literally running tens of thousands of these a day, so I thought I'd scale it back to just a few hundred, or a few thousand.

What do you think might be the problem? Do you think it might be a cocaine thing? I guess I don't really know where to start!

Kind regards,

(and thanks for your help, it really is much appreciated)

Geoff

On Tue, Sep 18, 2012 at 6:16 PM, Geoff Wright geoff@weartolook.comwrote:

Hi Jon,

Sorry, I've been away. I'll give it a go right now.

Thanks,

Geoff

On Fri, Sep 14, 2012 at 3:47 PM, Jon Yurek notifications@github.comwrote:

Hi, have you tried this new branch or the latest version of Paperclip to see if ti works?

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8563438.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL — Reply to this email directly or view it on GitHub.

Jonathan Yurek, Co-founder thoughtbot, inc. 617.482.1300 x114 http://thoughtbot.com/ http://twitter.com/thoughtbot

geoffw8 commented 12 years ago

Thanks, its very much appreciated.

Just to clarify, I literally just pop "Cocaine::CommandLine.runner = Cocaine::CommandLine::BackticksRunner.new" into initializers/cocaine.rb

(All the others seem to define a block)

And by run everything I need to you mean, it should sort this out? Or at least undo the more serious error I'm seeing?

Either way, thanks :)

jyurek commented 12 years ago

Yeah, just pop that line in there and it should work. And, yes, it should sort it out. It will make Cocaine use the runner that it used before all that got extracted, same as 0.2.7.

geoffw8 commented 12 years ago

Hey man, sorry to be a drain!

I keep getting:

\ [out :: mysite.com] rake aborted!

\ [out :: mysite.com] undefined method `runner=' for

Cocaine::CommandLine:Class

\ [out :: mysite.com]

\ [out :: mysite.com] Tasks: TOP => environment

\ [out :: mysite.com](See full trace by running task with --trace)

This is happening as its trying to precompile the assets. I did remove the initializer and deploy thinking that maybe the bundle was happening after this, so the gem hadn't updated but still no luck.

I have in my Gemfile.lock:

cocaine (0.3.0)

and in config/initializers/cocaine.rb

Cocaine::CommandLine.runner = Cocaine::CommandLine::BackticksRunner.new

Happy to just revert to the previous ref I had in there before if its easier.

Thanks,

Geoff

On Wed, Sep 19, 2012 at 4:31 PM, Jon Yurek notifications@github.com wrote:

Yeah, just pop that line in there and it should work. And, yes, it should sort it out. It will make Cocaine use the runner that it used before all that got extracted, same as 0.2.7.

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8694896.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

jyurek commented 12 years ago

Sorry I thought I mentioned, but you have to get what's on master. This code isn't released yet.

On Sep 19, 2012, at 12:02 PM, geoffw8 wrote:

Hey man, sorry to be a drain!

I keep getting:

\ [out :: mysite.com] rake aborted!

\ [out :: mysite.com] undefined method `runner=' for

Cocaine::CommandLine:Class

\ [out :: mysite.com]

\ [out :: mysite.com] Tasks: TOP => environment

\ [out :: mysite.com](See full trace by running task with --trace)

This is happening as its trying to precompile the assets. I did remove the initializer and deploy thinking that maybe the bundle was happening after this, so the gem hadn't updated but still no luck.

I have in my Gemfile.lock:

cocaine (0.3.0)

and in config/initializers/cocaine.rb

Cocaine::CommandLine.runner = Cocaine::CommandLine::BackticksRunner.new

Happy to just revert to the previous ref I had in there before if its easier.

Thanks,

Geoff

On Wed, Sep 19, 2012 at 4:31 PM, Jon Yurek notifications@github.com wrote:

Yeah, just pop that line in there and it should work. And, yes, it should sort it out. It will make Cocaine use the runner that it used before all that got extracted, same as 0.2.7.

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8694896.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL — Reply to this email directly or view it on GitHub.

Jonathan Yurek, Co-founder thoughtbot, inc. 617.482.1300 x114 http://thoughtbot.com/ http://twitter.com/thoughtbot

geoffw8 commented 12 years ago

Ahh sorry. You did. I was on a custom ref before, I thought you were just saying to go back to the main gem (I thought master was the main gem).

Cheers

On Wed, Sep 19, 2012 at 6:20 PM, Jon Yurek notifications@github.com wrote:

Sorry I thought I mentioned, but you have to get what's on master. This code isn't released yet.

On Sep 19, 2012, at 12:02 PM, geoffw8 wrote:

Hey man, sorry to be a drain!

I keep getting:

\ [out :: mysite.com] rake aborted!

\ [out :: mysite.com] undefined method `runner=' for

Cocaine::CommandLine:Class

\ [out :: mysite.com]

\ [out :: mysite.com] Tasks: TOP => environment

\ [out :: mysite.com](See full trace by running task with --trace)

This is happening as its trying to precompile the assets. I did remove the initializer and deploy thinking that maybe the bundle was happening after this, so the gem hadn't updated but still no luck.

I have in my Gemfile.lock:

cocaine (0.3.0)

and in config/initializers/cocaine.rb

Cocaine::CommandLine.runner = Cocaine::CommandLine::BackticksRunner.new

Happy to just revert to the previous ref I had in there before if its easier.

Thanks,

Geoff

On Wed, Sep 19, 2012 at 4:31 PM, Jon Yurek notifications@github.com wrote:

Yeah, just pop that line in there and it should work. And, yes, it should sort it out. It will make Cocaine use the runner that it used before all that got extracted, same as 0.2.7.

— Reply to this email directly or view it on GitHub< https://github.com/thoughtbot/cocaine/issues/23#issuecomment-8694896>.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL — Reply to this email directly or view it on GitHub.

Jonathan Yurek, Co-founder thoughtbot, inc. 617.482.1300 x114 http://thoughtbot.com/ http://twitter.com/thoughtbot

—

Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8698811.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

geoffw8 commented 12 years ago

Hi Jon,

Well, this doesn't seem to have helped, I'm now in fact in a pretty horrendous situation, I can't get more than 30 images to run before it stops dead at 100% CPU.

Am I the only one reporting this? I'd have thought anyone whose using the latest version/master would be having this issue. I've so far used every suggestion, I used the old branch, the new branch, with the initialiser, without and now I'm back to using the master branch with the initialiser.

I have no idea what to do. All of my other jobs run fine, I'm almost certain its a cocaine thing.

*Argument list too long - identify -format %wx%h

'/tmp/stream20120919-29345-bnq7j1[0]*

/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/core_ext/kernel/agnostics.rb:7:in'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/core_ext/kernel/agnostics.rb:7:in'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/bundler/gems/cocaine-06835ec16457/lib/cocaine/command_line/runners/backticks_runner.rb:7:inblock in call'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/bundler/gems/cocaine-06835ec16457/lib/cocaine/command_line/runners/backticks_runner.rb:17:inwith_modified_environment'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/bundler/gems/cocaine-06835ec16457/lib/cocaine/command_line/runners/backticks_runner.rb:6:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/bundler/gems/cocaine-06835ec16457/lib/cocaine/command_line.rb:89:inexecute'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/bundler/gems/cocaine-06835ec16457/lib/cocaine/command_line.rb:67:inrun'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip.rb:102:inrun'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/geometry.rb:22:infrom_file'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/thumbnail.rb:35:ininitialize'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/processor.rb:33:innew'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/processor.rb:33:inmake'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:441:inblock in post_process_style'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:440:ineach'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:440:ininject'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:440:inpost_process_style'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:433:inblock in post_process_styles'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:432:ineach'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:432:inpost_process_styles'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:425:inblock (2 levels) in post_process'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:390:in_run_photo_post_process_callbacks'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:81:inrun_callbacks'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/callback_compatibility.rb:54:inrun_paperclip_callbacks'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:424:inblock in post_process'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:399:in_run_post_process_callbacks'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/callbacks.rb:81:inrun_callbacks'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/callback_compatibility.rb:54:inrun_paperclip_callbacks'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:423:inpost_process'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip/attachment.rb:120:inassign'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/paperclip-2.7.0/lib/paperclip.rb:349:inblock in has_attached_file'\n/var/www/sites/ weartolook.com/releases/20120919205441/app/models/product.rb:616:inhard_get_photo'\n/var/www/sites/ weartolook.com/releases/20120919205441/app/workers/get_photo.rb:10:inperform'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/processor.rb:43:inblock (3 levels) in process'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:79:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:79:inblock in invoke'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/server/timeout.rb:14:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:81:inblock in invoke'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/server/active_record.rb:6:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:81:inblock in invoke'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/server/retry_jobs.rb:42:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:81:inblock in invoke'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/server/logging.rb:11:inblock in call'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/logging.rb:22:inwith_context'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/server/logging.rb:7:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:81:inblock in invoke'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:84:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/middleware/chain.rb:84:ininvoke'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/processor.rb:42:inblock (2 levels) in process'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/processor.rb:77:instats'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/sidekiq-2.3.1/lib/sidekiq/processor.rb:41:inblock in process'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/celluloid-0.12.0/lib/celluloid/calls.rb:57:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/celluloid-0.12.0/lib/celluloid/calls.rb:57:indispatch'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/celluloid-0.12.0/lib/celluloid/future.rb:18:inblock in initialize'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/celluloid-0.12.0/lib/celluloid/internal_pool.rb:48:incall'\n/var/www/sites/ weartolook.com/shared/bundle/ruby/1.9.1/gems/celluloid-0.12.0/lib/celluloid/internal_pool.rb:48:in`block in create'

Geoff

On Wed, Sep 19, 2012 at 6:43 PM, Geoff Wright geoff@weartolook.com wrote:

Ahh sorry. You did. I was on a custom ref before, I thought you were just saying to go back to the main gem (I thought master was the main gem).

Cheers

On Wed, Sep 19, 2012 at 6:20 PM, Jon Yurek notifications@github.comwrote:

Sorry I thought I mentioned, but you have to get what's on master. This code isn't released yet.

On Sep 19, 2012, at 12:02 PM, geoffw8 wrote:

Hey man, sorry to be a drain!

I keep getting:

\ [out :: mysite.com] rake aborted!

\ [out :: mysite.com] undefined method `runner=' for

Cocaine::CommandLine:Class

\ [out :: mysite.com]

\ [out :: mysite.com] Tasks: TOP => environment

\ [out :: mysite.com](See full trace by running task with --trace)

This is happening as its trying to precompile the assets. I did remove the initializer and deploy thinking that maybe the bundle was happening after this, so the gem hadn't updated but still no luck.

I have in my Gemfile.lock:

cocaine (0.3.0)

and in config/initializers/cocaine.rb

Cocaine::CommandLine.runner = Cocaine::CommandLine::BackticksRunner.new

Happy to just revert to the previous ref I had in there before if its easier.

Thanks,

Geoff

On Wed, Sep 19, 2012 at 4:31 PM, Jon Yurek notifications@github.com wrote:

Yeah, just pop that line in there and it should work. And, yes, it should sort it out. It will make Cocaine use the runner that it used before all that got extracted, same as 0.2.7.

— Reply to this email directly or view it on GitHub< https://github.com/thoughtbot/cocaine/issues/23#issuecomment-8694896>.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL — Reply to this email directly or view it on GitHub.

Jonathan Yurek, Co-founder thoughtbot, inc. 617.482.1300 x114 http://thoughtbot.com/ http://twitter.com/thoughtbot

—

Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8698811.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

jyurek commented 12 years ago

Well, now your problem is the same as the Paperclip thread in that you seem to be getting E2BIG errors -- so, yes, others are seeing this same problem now. But what doesn't make sense is that you ay you're getting 100% CPU usage and errors. That doesn't seem possible, since if you get the error then it should be stopping. Unless it just goes right on to the next one where it hangs.

Out of curiosity, the other jobs you mention, are they running command line utilities, or are they pure-ruby rake tasks or something?

Have you turned on the Cocaine logger to see what commands, exactly, are being run? You can use Cocaine::CommandLine.logger = Rails.logger to put the Cocaine commands into the Rails log.

I apologize for the hassle here. None of this makes sense to me, either. I can't find anything that looks like it should be failing for you.

jyurek commented 12 years ago

@geoffw8, a though occurs: You started this issue with the heading of "Multi-threading support"... are you actually running multiple Threads in one ruby process, or are you running the workers as separate processes?

geoffw8 commented 12 years ago

Hi Jon,

I'll turn logging on in a moment, and to my knowledge I am using multiple threads in one ruby process - I use sidekiq ( http://mperham.github.com/sidekiq/).

Thanks for all your attention on this.

Cheers,

Geoff

On Fri, Sep 21, 2012 at 3:47 PM, Jon Yurek notifications@github.com wrote:

@geoffw8 https://github.com/geoffw8, a though occurs: You started this issue with the heading of "Multi-threading support"... are you actually running multiple Threads in one ruby process, or are you running the workers as separate processes?

— Reply to this email directly or view it on GitHubhttps://github.com/thoughtbot/cocaine/issues/23#issuecomment-8766988.

BR,

Geoff Wright Founder & CEO

w: weartolook.com t: @weartolook http://www.twitter.com/weartolook

weartolook LTD St John's House, 54 St John's Square, London, EC1V4JL

jyurek commented 12 years ago

Ok, while that's annoying it might be the cause of the problem. Can you try running the processing in a single thread, if possible?

jyurek commented 12 years ago

@geoffw8 I've fixed a problem in another issue that may possibly have ramifications here. Can you try specifying cocaine as gem 'cocaine', :git => "git://github.com/thoughtbot/cocaine.git", :branch => "v0.3" in your Gemfile?

jyurek commented 12 years ago

@geoffw8 I've released these changes as v0.3.2, if you'd like to try that out and see if it fixes your problem.

jyurek commented 12 years ago

I'm going to close this. If it pops back up or it's still a problem please let me know.