Converter tools cannot locate job_resource_params_conf.xml file 20.09.rc1

[code4dna]

We have the following recently installed version:

commit 1cc3d5b17820829de6dcd9ce054e8e797cf37d8a
Date:   Wed Sep 30 11:43:49 2020 -0400

The job resource selector (cores and time) is working perfect for default tools and these toolshed tools: Trimmomatic, SPAdes, FastQC and 'Compute Quality Statistics'. The toolshed tools bwa, bwa-mem, bowtie2 and minimap2 fail immediately with the following pop up error message after clicking Execute without being scheduled to run.

The server could not complete the request. Please contact the Galaxy Team if this error persists. 
Error executing tool with id 'toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2': 'cores'

{
    "history_id": "303905fee1757a23",
    "tool_id": "toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2",
    "tool_version": "2.17+galaxy2",
    "inputs": {
        "reference_source|reference_source_selector": "history",
        "reference_source|ref_file": {
            "values": [
                {
                    "id": "ecc8650bee0bd5b4",
                    "hid": 83,
                    "name": "http://candidagenome.org/download/sequence/C_dubliniensis_CD36/current/C_dubliniensis_CD36_current_chromosomes.fasta.gz (as fasta)",
                    "tags": [],
                    "src": "hda",
                    "keep": false
                }
            ],
            "batch": false
        },
        "fastq_input|fastq_input_selector": "single",
        "fastq_input|fastq_input1": {
            "values": [
                {
                    "id": "fd44a526bec7f18e",
                    "hid": 93,
                    "name": "Trimmomatic on DR34_R1.fastq.gz",
                    "tags": [],
                    "src": "hda",
                    "keep": false
                }
            ],
            "batch": false
        },
        "fastq_input|analysis_type_selector": null,
        "__job_resource|__job_resource__select": "yes",
        "__job_resource|cores": "28",
        "__job_resource|time": 24
    }
}

Here is the galaxy.log

galaxy.tools ERROR 2020-10-02 10:33:20,994 [p:14906,w:1,m:0] [uWSGIWorker1Core3] Exception caught while attempting to execute tool with id 'toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2':
Traceback (most recent call last):
  File "lib/galaxy/tools/__init__.py", line 1647, in handle_single_execution
    flush_job=flush_job,
  File "lib/galaxy/tools/__init__.py", line 1736, in execute
    return self.tool_action.execute(self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs)
  File "lib/galaxy/tools/actions/__init__.py", line 292, in execute
    history, inp_data, inp_dataset_collections, preserved_tags, all_permissions = self._collect_inputs(tool, trans, incoming, history, current_user_roles, collection_info)
  File "lib/galaxy/tools/actions/__init__.py", line 259, in _collect_inputs
    inp_data, all_permissions = self._collect_input_datasets(tool, incoming, trans, history=history, current_user_roles=current_user_roles, collection_info=collection_info)
  File "lib/galaxy/tools/actions/__init__.py", line 202, in _collect_input_datasets
    tool.visit_inputs(param_values, visitor)
  File "lib/galaxy/tools/__init__.py", line 1523, in visit_inputs
    visit_input_values(self.inputs, values, callback)
  File "lib/galaxy/tools/parameters/__init__.py", line 164, in visit_input_values
    visit_input_values(input.cases[values['__current_case__']].inputs, values, callback, new_name_prefix, label_prefix, parent_prefix=name_prefix, **payload)
  File "lib/galaxy/tools/parameters/__init__.py", line 170, in visit_input_values
    callback_helper(input, input_values, name_prefix, label_prefix, parent_prefix=parent_prefix, context=context)
  File "lib/galaxy/tools/parameters/__init__.py", line 131, in callback_helper
    new_value = callback(**args)
  File "lib/galaxy/tools/actions/__init__.py", line 144, in visitor
    input_datasets[prefix + input.name] = process_dataset(value)
  File "lib/galaxy/tools/actions/__init__.py", line 96, in process_dataset
    data = data.get_converted_dataset(trans, target_ext, target_context=parent, history=history)
  File "lib/galaxy/model/__init__.py", line 2876, in get_converted_dataset
    new_dataset = next(iter(self.datatype.convert_dataset(trans, self, target_ext, return_output=True, visible=False, deps=deps, target_context=target_context, history=history).values()))
  File "lib/galaxy/datatypes/data.py", line 678, in convert_dataset
    converted_dataset = converter.execute(trans, incoming=params, set_output_hid=visible, history=history)[1]
  File "lib/galaxy/tools/__init__.py", line 1736, in execute
    return self.tool_action.execute(self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs)
  File "lib/galaxy/tools/actions/__init__.py", line 354, in execute
    params=wrapped_params.params,
  File "lib/galaxy/tools/parameters/wrapped.py", line 44, in params
    self.wrap_values(self.tool.inputs, params, skip_missing_values=not self.tool.check_values)
  File "lib/galaxy/tools/parameters/wrapped.py", line 68, in wrap_values
    self.wrap_values(input.cases[current].inputs, values, skip_missing_values=skip_missing_values)
  File "lib/galaxy/tools/parameters/wrapped.py", line 59, in wrap_values
    value = input_values[input.name]
KeyError: 'cores'
galaxy.tools.execute WARNING 2020-10-02 10:33:20,995 [p:14906,w:1,m:0] [uWSGIWorker1Core3] There was a failure executing a job for tool [toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2] - Error executing tool with id 'toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2': 'cores'

[pcm32]

I have seen this in the past, for instance with the alevin tool from @bgruening I think. In the end we had to take out the resources definition, would be good to nail it to be able to get resources selection back!

[pcm32]

I have seen this with versions as old as 19.05 or even older possibly.

[code4dna]

The toolshed minimap2 works fine in our 20.05 installation with job resource params so I'll be able to compare the two installations. This may be related to #10267 I've tried each of the following values in galaxy.yml for our latest install but they all produce the same error:

#job_resource_params_file: job_resource_params_conf.xml

job_resource_params_file: job_resource_params_conf.xml

job_resource_params_file: config/job_resource_params_conf.xml

[code4dna]

I checked out the commit for today's usegalaxy.org but the error still occurs.

commit 04882252b3c87ea31f2979d1cc64ebe0f429c15a
Date:   Mon Sep 28 19:27:21 2020 +0200

I am using these same three files that work for 20.05:

lib/galaxy/jobs/rules/map_resources.py
config/job_conf.xml
config/job_resource_params_conf.xml

I notice in the galaxy.log that map_resources.py isn't run since there are no log messages from map_resources.py. It seems that some tools cannot find config/job_resource_params_conf.xml since the error message is similar to #10267

[jmchilton]

This would appear to be the same problem as https://github.com/galaxyproject/galaxy/issues/9599. No clue what is causing it though. Could it be an issue with datatypes and implicit converters?

[code4dna]

Looks like it is related to #9599 and was fixed for a while at least it works in our 20.05 install but now it fails in 20.09.rc1. The tools that are failing, bwa, bowtie2 and minimap2, all have an option to create a reference genome index from a fasta file in the history. If the reference genome sequence file is compressed (fasta.gz) then bwa, bowtie2 and minimap2 fail but if the reference genome is not compressed, then bwa, bowtie2 and minimap2 will align successfully without errors even if the sample fastq files are compressed.

[code4dna]

In 20.05 and 20.09.rc1, the Edit attributes: Convert datatype from compressed file to uncompressed fails with a short message:

__sq__cores__sq__

If I switch the group_id order in job_conf.xml from cores,time to time,cores

    <resources default="cores_time">
      <group id="cores_time">time,cores</group>
    </resources>

Then the error message is: __sq__time__sq__

Although the alignment tools are working with fasta.gz reference genomes for indexing in 20.05.

[code4dna]

It seems that the format converters are looking for the job_resource_params_conf.xml file but can't find it since the error message is similar to #10267 which is the error message when the job_resource_params_conf.xml file is not found. It also seems that the converters do not need to look for and use the job_resource_params_conf.xml, except maybe if the conversion needs to occur on a remote configured system during a job. If the converters searching for the job_resource_params_conf.xml file can be disabled, it may fix this issue at least for cores, time, memory resources.

[jdavcs]

I'm looking into #10267 at the moment; this may or may not be related to a config issue (job_resource_params_file shouldn't have to be prefixed with config/ in 20.09).

[code4dna]

I can accurately define this issue now. We have job_conf.xml configured to enable 'Job Resource Parameters' on every tool:

    <resources default="cores_time">
      <group id="cores_time">cores,time</group>
    </resources>

This way we don't have to add a line to job_conf.xml for each newly installed tool that we want to use resource params. Using this approach also causes the converter tools to look for the job_resource_params_conf.xml which they can't locate which is what is failing. As a test, I changed our config to allow 'Job Resource Parameters' on just the bwa tool and all works well and bwa will run successfully using job resource params when the genome fasta file is compressed. Additionally, converting a file datatype from compressed file to uncompressed works.

        <destination id="dest_cores_time" runner="dynamic">
            <param id="type">python</param>
            <param id="function">dynamic_cores_time</param>
        </destination>

   <tools>
    <tool id="bwa" destination="dest_cores_time" resources="cores_time"/>
   </tools>
    <resources default="default">
      <group id="default"></group>
      <group id="cores_time">cores,time</group>
    </resources>

In a separate test, I set the lib/galaxy/datatypes/converters/gz_to_uncompressed.xml tool to use local destination instead of the default resource params but it fails. <tool id="CONVERTER_gz_to_uncompressed" destination="local"/> There may be other converters/tools used when an input file needs to be uncompressed.

[pcm32]

The tools that are failing, bwa, bowtie2 and minimap2, all have an option to create a reference genome index from a fasta file in the history.

The alevin tool that was failing for us also had that. But then I think we found other tools failing in a similar way that were not mappers.

[code4dna]

I renamed this issue since it is related with the converters or other metadata? tools unable to locate the job_resource_params_conf.xml file and not with the toolshed tools' configs. I think the converters do not need to use the job_resource_params_conf.xml file at least for cores, time, memory.