search

google-code-export / yabi

Automatically exported from code.google.com/p/yabi
0 stars 1 forks source link

BE doesn't update status of tasks for large workflow #184

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. On next_release branch run the quickstart tests (fab runtests) or the 
"nosetests -v 
simple_tool_tests.py:ExplodingBackendTest.test_submit_json_directly_larger_workf
low"
2. The test is creating a workflow containing 50 hostname jobs to be run 
against the Exploding Backend

What is the expected output? What do you see instead?

The test should pass, but instead it hangs, because some of the tasks stay on 
the status "exec:running".
I've added extra logging statements and it seems that for some of the tasks we 
simply don't receive a call to the status webservice requesting to set the 
status to error.

Original issue reported on code.google.com by szab...@gmail.com on 3 Apr 2012 at 8:10

GoogleCodeExporter commented 9 years ago 
I can't replicate on ubuntu 11.04 or 11.10. I also know Andrew cannot replicate 
on ubuntu 10.10

Original comment by aahun...@gmail.com on 4 Apr 2012 at 6:01

GoogleCodeExporter commented 9 years ago 
ok by making the speed bomb test have a delay of zero I can now see this error.

The changeset:

http://code.google.com/p/yabi/source/detail?r=0bc12d0dc72f54cc1cee636a3fc9fa4ed2
7725ca&name=next_release

Attached is a grep of admin log for the test:

grep "status=" yabiadmin/yabiadmin/yabiadmin.log >> 
hostname.100.exploding.backend.nodelay.txt

In admin, all the tasks are stuck on exec:running or exec:unsubmitted.

Also attached is the yabibe log for the test run.

Twisted is using 100% CPU (on one core) and has been for about 30 minutes. 
Nothing is hitting admin, nothing going in twisted logs.

Original comment by aahun...@gmail.com on 5 Apr 2012 at 5:27

Attachments:

GoogleCodeExporter commented 9 years ago 
screenshot of admin

Original comment by aahun...@gmail.com on 5 Apr 2012 at 5:30

Attachments:

GoogleCodeExporter commented 9 years ago 
Updated twisted, greenlet and gevent (from source) and same result.

(virt_yabibe)ahunter@ubuntu:~/devel/gc/yabi/yabibe/yabibe$ pip freeze
Fabric==1.2.2
Mako==0.4.2
MarkupSafe==0.9.3
Twisted==12.0.0
TwistedWeb2==10.2.0
boto==2.2.2
ccg-python-build==2.2.9
gevent==1.0dev
greenlet==0.3.4
mercurial==1.9.3
paramiko==1.7.7.1
psutil==0.4.1
pyOpenSSL==0.13
pycrypto==2.3
setproctitle==1.1.2
virtualenv==1.6.4
wsgiref==0.1.2
zope.interface==3.8.0

Original comment by aahun...@gmail.com on 5 Apr 2012 at 6:37

GoogleCodeExporter commented 9 years ago

Original comment by szab...@gmail.com on 11 Apr 2012 at 12:53

GoogleCodeExporter commented 9 years ago 
Fixed in d6a65749d155 on branch tszabo.

Original comment by szab...@gmail.com on 12 Apr 2012 at 3:26