search

documentcloud / cloud-crowd

Parallel Processing for the Rest of Us
https://github.com/documentcloud/cloud-crowd/wiki
MIT License
851 stars 92 forks source link

job using graphics_magick.rb action fails #19

Closed jbfink closed 14 years ago

jbfink commented 14 years ago

I've managed to get a modest cloud running of two nodes using the cloud-crowd 0.5.0 gem, sqlite db, and a local filesystem shared over NFS. The Shakespeare wordcount job runs just fine in this small cloud; however, when I try to run a job with the included graphics_magick.rb action, everything gets distributed out to the nodes okay, the nodes loads spike in the way that you'd expect (the job involves transforming gigantic TIFF files to JPEG2000) but the job never actually completes. Does anyone have experience running jobs with graphics_magick.rb? Here's the (possibly useless) node.log from one of the nodes -- http://gist.github.com/449965 -- and there's a somewhat truncated version of the job file we're using -- http://gist.github.com/449976.

Anything jump out at you looking at this? I'm pretty stumped. Thanks for the help.

jashkenas commented 14 years ago

Looks from the log like the GraphicsMagick command is failing.

I'm afraid that the included example actions are just that -- examples, and aren't meant for serious work or debugging. I'd recommend you write a version of the GraphicsMagick action that does the job you're looking for, and logs better errors (or the entire output of gm) while it's at it.

In general, for debugging actions, it's good to test them in isolation before deploying them to a cluster. If you do need to debug them while they're running live, node.log is your best bet.