search

hmenager / workflow-is-cwl

This repository contains CWL descriptions of the various tools which will allow you to build workflows for the annotation of transcripts
https://www.elixir-europe.org/
0 stars 0 forks source link

sync201811 - galaxy+cwltool - cmsearch-multimodel-wf #23

Open ghost opened 5 years ago

ghost commented 5 years ago 
$ cwltool --pack cmsearch-multimodel-wf.cwl > cmsearch-multimodel-wf-packed.cwl

When importing the packed workflow in Galaxy, error below occurs

Traceback (most recent call last):
  File "lib/galaxy/web/framework/decorators.py", line 283, in decorator
    rval = func(self, trans, *args, **kwargs)
  File "lib/galaxy/webapps/galaxy/api/workflows.py", line 337, in create
    return self.__api_import_from_archive(trans, archive_data, source="uploaded file", from_path=os.path.abspath(uploaded_file_name))
  File "lib/galaxy/webapps/galaxy/api/workflows.py", line 596, in __api_import_from_archive
    raw_workflow_description = self.__normalize_workflow(trans, data)
  File "lib/galaxy/webapps/galaxy/api/workflows.py", line 679, in __normalize_workflow
    return self.workflow_contents_manager.normalize_workflow_format(trans, as_dict)
  File "lib/galaxy/managers/workflows.py", line 315, in normalize_workflow_format
    workflow_path += "#" + object_id
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'
ghost commented 5 years ago

Fixed with code below

      def __api_import_from_archive(self, trans, archive_data, source=None, from_path=None):
          try:
              data = json.loads(archive_data)
              if from_path is not None:
                  data.update({"src": "from_path", "path": from_path})
          except Exception:
              raise exceptions.MessageException("The data content does not appear to be a valid workflow.")
ghost commented 5 years ago

We can now see the workflow in the Galaxy workflow editor, but when we click on "Save" to persist modifications, error below occurs

Traceback (most recent call last):
  File "lib/galaxy/web/framework/decorators.py", line 283, in decorator
    rval = func(self, trans, *args, **kwargs)
  File "lib/galaxy/webapps/galaxy/api/workflows.py", line 552, in update
    **from_dict_kwds
  File "lib/galaxy/managers/workflows.py", line 425, in update_workflow_from_raw_description
    self._sync_stored_workflow(trans, stored_workflow)
  File "lib/galaxy/managers/workflows.py", line 540, in _sync_stored_workflow
    wf_dict = from_galaxy_native(wf_dict, None, json_wrapper=True)
  File "/home/jra001k/snapshot/galaxy/.venv/local/lib/python2.7/site-packages/gxformat2/export.py", line 48, in from_galaxy_native
    source = _to_source(workflow_output, label_map, output_id=step["id"])
  File "/home/jra001k/snapshot/galaxy/.venv/local/lib/python2.7/site-packages/gxformat2/export.py", line 217, in _to_source
    output_id = output_id or has_output_name['id']
KeyError: 'id'
ghost commented 5 years ago

The problem seems cause by the line below, when output_id value si 0

output_id = output_id or has_output_name['id']

Fixed in .venv/lib/python2.7/site-packages/gxformat2/export.py with patch below

Before

def _to_source(has_output_name, label_map, output_id=None):
    output_id = output_id or has_output_name['id']

After

def _to_source(has_output_name, label_map, output_id=None):
    output_id = output_id if output_id is not None else has_output_name['id']
ghost commented 5 years ago

When we restart Galaxy (after having edited and saved the workflow), and we try to edit the wf again, warning below occurs

galaxy.workflow.modules WARNING 2018-12-11 06:15:52,195 [p:16652,w:1,m:0] [uWSGIWorker1Core0] The tool 'tmpugCygk#infernal-cmsearch-v1.1.2.cwl' is missing. Cannot build workflow module.                                       
galaxy.workflow.modules WARNING 2018-12-11 06:15:52,210 [p:16652,w:1,m:0] [uWSGIWorker1Core0] The tool 'tmpugCygk#concatenate.cwl' is missing. Cannot build workflow module.                                                    
galaxy.workflow.modules WARNING 2018-12-11 06:15:52,214 [p:16652,w:1,m:0] [uWSGIWorker1Core0] The tool 'tmpugCygk#cmsearch-deoverlap-v0.02.cwl' is missing. Cannot build workflow module.

and the wf cannot be edited anymore (links and boxes are missing).

It seems to be caused by the following change in the database: in workflow_step table, tool are now referenced using tool_id column (e.g. 'tmpPd1pDz#cmsearch-deoverlap-v0.02.cwl') instead of tool_hash column.

Before saving:

sqlite> select type,label,tool_id,tool_hash from workflow_step where type='tool' order by label; 
tool        cmsearch                7943013e61897f4949e77546edeabd960a2f608067e2ecd523a1d2ef010e9682
tool        concatenat              742a9252f4cf1b80bcb57d20aa4133edbaa46b8a3112a9055110db18b1e446dc
tool        remove_ove              e384c3b53464dbea34880afedfb09f6857ab89313014f76f9eea23e15e1c83b8

After saving:

sqlite> select type,label,tool_id,tool_hash from workflow_step where type='tool' order by label; 
tool        cmsearch                7943013e61897f4949e77546edeabd960a2f608067e2ecd523a1d2ef010e9682
tool        cmsearch    tmpkadxtU#                                                                  
tool        concatenat              742a9252f4cf1b80bcb57d20aa4133edbaa46b8a3112a9055110db18b1e446dc
tool        concatenat  tmpkadxtU#                                                                  
tool        remove_ove              e384c3b53464dbea34880afedfb09f6857ab89313014f76f9eea23e15e1c83b8
tool        remove_ove  tmpkadxtU#
ghost commented 5 years ago

_

ghost commented 5 years ago

The workflow run successfully (as long as it has not been edited and saved with the Galaxy editor (see above)) with parameters below:

omit_alignment_section: true
only_hmm: true
search_space_size: 1000

If the workflow runs after having been edited and saved with the Galaxy editor, error below occurs:

Traceback (most recent call last):
  File "lib/galaxy/workflow/run.py", line 192, in invoke
    incomplete_or_none = self._invoke_step(workflow_invocation_step)
  File "lib/galaxy/workflow/run.py", line 268, in _invoke_step
    use_cached_job=self.workflow_invocation.use_cached_job)
  File "lib/galaxy/workflow/modules.py", line 721, in execute
    progress.set_outputs_for_input(invocation_step, step_outputs)
  File "lib/galaxy/workflow/run.py", line 567, in set_outputs_for_input
    self.set_step_outputs(invocation_step, outputs)
  File "lib/galaxy/workflow/run.py", line 598, in set_step_outputs
    output=output,
  File "lib/galaxy/workflow/run.py", line 602, in _record_workflow_output
    self.workflow_invocation.add_output(workflow_output, step, output)
  File "lib/galaxy/model/__init__.py", line 4504, in add_output
    if output_object.history_content_type == "dataset":
AttributeError: 'NoneType' object has no attribute 'history_content_type'
galaxy.workflow.run ERROR 2018-11-29 06:59:48,291 [p:1567,w:1,m:0] [WorkflowRequestMonitor.monitor_thread] Failed to execute scheduled workflow.
Traceback (most recent call last):
  File "lib/galaxy/workflow/run.py", line 85, in __invoke
    outputs = invoker.invoke()
  File "lib/galaxy/workflow/run.py", line 192, in invoke
    incomplete_or_none = self._invoke_step(workflow_invocation_step)
  File "lib/galaxy/workflow/run.py", line 268, in _invoke_step
    use_cached_job=self.workflow_invocation.use_cached_job)
  File "lib/galaxy/workflow/modules.py", line 721, in execute
    progress.set_outputs_for_input(invocation_step, step_outputs)
  File "lib/galaxy/workflow/run.py", line 567, in set_outputs_for_input
    self.set_step_outputs(invocation_step, outputs)
  File "lib/galaxy/workflow/run.py", line 598, in set_step_outputs
    output=output,
  File "lib/galaxy/workflow/run.py", line 602, in _record_workflow_output
    self.workflow_invocation.add_output(workflow_output, step, output)
  File "lib/galaxy/model/__init__.py", line 4504, in add_output
    if output_object.history_content_type == "dataset":
AttributeError: 'NoneType' object has no attribute 'history_content_type'