search

Illumina / pyflow

A lightweight parallel task engine
http://Illumina.github.io/pyflow/
145 stars 44 forks source link

v1.1.4 causes problems when self.getRunMode() is called from a WorkflowTask #5

Closed hyjkim closed 9 years ago

hyjkim commented 9 years ago

Calling self.getRunMode() in a nested workflow results in the following error. The error does not occur when using v1.1.3.

[2015-07-07T19:14:27.835594] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] Worklow terminated due to the following task errors:
[2015-07-07T19:14:27.836939] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] Failed to complete sub-workflow task: 'set' launched from master workflow, failed sub-workflow classname: 'Setter'
[2015-07-07T19:14:27.837627] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set] Error Message:
[2015-07-07T19:14:27.838230] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set] Unhandled Exception in TaskRunner-Thread-set
[2015-07-07T19:14:27.838926] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set] Traceback (most recent call last):
[2015-07-07T19:14:27.839591] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]   File "/illumina/scratch/panacea/users/hkim/virtualenv/pyflow_v1.1.4/lib/python2.7/site-packages/pyflow/pyflow.py", line 1007, in run
[2015-07-07T19:14:27.840229] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]     (retval, retmsg) = self._run()
[2015-07-07T19:14:27.840842] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]   File "/illumina/scratch/panacea/users/hkim/virtualenv/pyflow_v1.1.4/lib/python2.7/site-packages/pyflow/pyflow.py", line 1059, in _run
[2015-07-07T19:14:27.841444] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]     self.workflow.workflow()
[2015-07-07T19:14:27.842045] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]   File "runmode.py", line 8, in workflow
[2015-07-07T19:14:27.842775] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]     self.flowLog("Setter Runmode: %s" % self.getRunMode()) # Logs "initial value"
[2015-07-07T19:14:27.843420] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]   File "/illumina/scratch/panacea/users/hkim/virtualenv/pyflow_v1.1.4/lib/python2.7/site-packages/pyflow/pyflow.py", line 3445, in getRunMode
[2015-07-07T19:14:27.844073] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]     self._requireInWorkflow()
[2015-07-07T19:14:27.844668] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]   File "/illumina/scratch/panacea/users/hkim/virtualenv/pyflow_v1.1.4/lib/python2.7/site-packages/pyflow/pyflow.py", line 4057, in _requireInWorkflow
[2015-07-07T19:14:27.845335] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set]     raise Exception("Method must be a (call stack) decendent of WorkflowRunner workflow() method (via run() method)")
[2015-07-07T19:14:27.846048] [ussf-prd-lndt-b-5-1.local] [2779_1] [WorkflowRunner] [ERROR] [set] Exception: Method must be a (call stack) decendent of WorkflowRunner workflow() method (via run() method)

Code to duplicate this error is below:

from pyflow import WorkflowRunner

class Child(WorkflowRunner):
    """
    Workflow called by master
    """
    def workflow(self):
        self.flowLog("Child Runmode: %s" % self.getRunMode())

class Master(WorkflowRunner):
    """
    Master workflow that calls Child
    """
    def workflow(self):
        child = Child()
        self.addWorkflowTask('set', child)
        self.flowLog("Master Runmode: %s" % self.getRunMode())

if __name__ == "__main__":
    master = Master()
    master.run()
isthisthat commented 9 years ago

Got the same error with a different call: (Works on 1.1.2 and 1.1.3 but not 1.1.4)

[WorkflowRunner] [ERROR] [ds] Traceback (most recent call last): [WorkflowRunner] [ERROR] [ds] File "/illumina/thirdparty/python/python-2.7.5/lib/python2.7/site-packages/pyflow/pyflow.py", line 1007, in run [WorkflowRunner] [ERROR] [ds](retval, retmsg) = self._run() [WorkflowRunner] [ERROR] [ds] File "/illumina/thirdparty/python/python-2.7.5/lib/python2.7/site-packages/pyflow/pyflow.py", line 1059, in _run [WorkflowRunner] [ERROR] [ds] self.workflow.workflow() [WorkflowRunner] [ERROR] [ds] File "/illumina/development/sharedCode/scripts/downsample.py", line 686, in workflow [WorkflowRunner] [ERROR] [ds] nCores = 8 if not self.isDryRun() else 1 [WorkflowRunner] [ERROR] [ds] File "/illumina/thirdparty/python/python-2.7.5/lib/python2.7/site-packages/pyflow/pyflow.py", line 3527, in isDryRun [WorkflowRunner] [ERROR] [ds] self._requireInWorkflow() [WorkflowRunner] [ERROR] [ds] File "/illumina/thirdparty/python/python-2.7.5/lib/python2.7/site-packages/pyflow/pyflow.py", line 4057, in _requireInWorkflow [WorkflowRunner] [ERROR] [ds] raise Exception("Method must be a (call stack) decendent of WorkflowRunner workflow() method (via run() method)") [WorkflowRunner] [ERROR] [ds] Exception: Method must be a (call stack) decendent of WorkflowRunner workflow() method (via run() method)

ctsa commented 9 years ago

Candidate fix is on master now. Can you check that this fixes your respective issues? If so, I can roll this as 1.1.5

hyjkim commented 9 years ago

Corrected the issue for me.