add test showing error during import job preparion with latest Galaxy

[bernt-matthias]

The following Traceback can be found in the logs (the test itself is not executed)

Traceback (most recent call last):
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/jobs/runners/__init__.py", line 291, in prepare_job
    job_wrapper.prepare()
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/jobs/__init__.py", line 1254, in prepare
    ) = tool_evaluator.build()
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/tools/evaluation.py", line 566, in build
    global_tool_logs(self._build_config_files, config_file, "Building Config Files")
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/tools/evaluation.py", line 96, in global_tool_logs
    raise e
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/tools/evaluation.py", line 92, in global_tool_logs
    return func()
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/tools/evaluation.py", line 633, in _build_config_files
    self.__write_workdir_file(config_filename, config_text, param_dict, is_template=is_template)
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/tools/evaluation.py", line 756, in __write_workdir_file
    value = fill_template(content, context=context, python_template_version=self.tool.python_template_version)
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/util/template.py", line 143, in fill_template
    raise first_exception or e
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/util/template.py", line 87, in fill_template
    return unicodify(t, log_exception=False)
  File "/tmp/tmpiottroee/galaxy-dev/lib/galaxy/util/__init__.py", line 1153, in unicodify
    value = str(value)
  File "/home/berntm/.planemo/gx_venv_3/lib/python3.10/site-packages/Cheetah/Template.py", line 1053, in __unicode__
    return getattr(self, mainMethName)()
  File "cheetah_DynamicallyCompiledCheetahTemplate_1705952649_1165519_57877.py", line 119, in respond
  File "cheetah_DynamicallyCompiledCheetahTemplate_1705952649_1165519_57877.py", line 102, in stringify
  File "cheetah_DynamicallyCompiledCheetahTemplate_1705952649_1165519_57877.py", line 113, in stringify
AttributeError: 'TreeDict' object has no attribute 'is_collection'

I also tried to understand what the config file is doing. If I get it right it generates a json (where some elements are stripped).

Did you know that a json file containing the inputs can also be generated with:

<configfiles>
    <inputs name="inputs" data_style='paths'/>
</configfiles>

see https://docs.galaxyproject.org/en/master/dev/schema.html#tool-configfiles-inputs. Guess stripping elements could be done with a little helper script that could run in the tool (before q2galaxy).

Seems that this fails with 23.1 but worked in 23.0. I will try to bisect this, but it's unlikely that this is a Galaxy bug ... the configfile just seems to access some Galaxy internals that may be subject to change.

I also would be a huge fan of having a few tests for the fastq import here.

In order to get the test running I needed to add collection_type="list" to the collection parameters. Might also be a good thing to add.

[bernt-matthias]

ping @ebolyen

[gregcaporaso]

@bernt-matthias, is this urgent on your end? @ebolyen is out till this Thursday, but I may be able to get ahold of him. Let me know.

[bernt-matthias]

is this urgent on your end?

not so urgent :) no worries.

Let me know if I can help :)

[gregcaporaso]

Thanks @bernt-matthias!

[Oddant1]

Ok. I am satisfied that this works and am now merging it as a stop gap until the next q2galaxy release makes more permanent changes. Thank you @bernt-matthias and @mvdbeek

[bernt-matthias]

I think this was not deployed. The test toolshed is currently not working. Probably you need to https://github.com/galaxyproject/tools-iuc/blob/0da1ca313d02ca780a671247e434a424fa67887b/.github/workflows/pr.yaml#L392

For this PR its probably the easiest to push manually to the toolshed.

[Oddant1]

@bernt-matthias I made that change, and it got us past the previous block, but there is now an issue with templating out one of the tools that I can't properly track down. I know where it's coming from, but I don't know what's causing it to happen (or honestly it's more like I don't know why it hasn't always been a problem). I'll most likely need to consult with @ebolyen about it when he comes in on Thursday since he knows the most about this code-base. I apologize for the delay.

[bernt-matthias]

If you point me to the error I might have a look.

I guess it would be sufficient to create another PR (introducing a mock change in the tool) and merge it. Then it should be deployed.

[Oddant1]

@bernt-matthias The issue we're seeing now doesn't seem to have anything to do with Galaxy itself it appears to be purely a q2galaxy thing to do with how the validator for the input for the quality-filter q-score action is being rendered. Previously the validator was checking if the type of the artifact provided was in the list:

['SampleData[JoinedSequencesWithQuality]', 'SampleData[SequencesWithQuality]', 'SampleData[PairedEndSequencesWithQuality]']

Now it is checking if the provided type is in the list:

['SampleData[JoinedSequencesWithQuality]', 'SampleData[SequencesWithQuality | PairedEndSequencesWithQuality]']

Causing an artifact of the type 'SampleData[SequencesWithQuality]' to fail the validator. This is the test that is failing in CI and when I render the tools locally I also get the later of the two validators. The strange thing is that even when I go back to older versions of QIIME 2 and q2galaxy I still get the bottom validator.

The bottom one is more accurate to how the type expression is written inside of q2-quality-filter, but the tool for q-score that currently lives in this repo has the first validator, and I can't for the life of me figure out how we ended up getting that first validator into the tool when the second one is all I can get rendered on any version of things, and the relevant code has not changed for some time, and the second one is a more accurate representation of the type used in the actual underlying QIIME 2 action.

RE: mock changing the tool, that's just the thing the existing tool in this repo works (though I have no idea how it was templated in the way it is) it's the one that's being re-rendered in this CI action that fails, so we are unable to re-render the tools. CI has gotten past the action it was failing on yesterday. I'm not sure if you need all the tools to re-render to be able to do what you need to do to deploy the import you manually modified. I suspect not now that I think about it.

[bernt-matthias]

I'm not sure if you need all the tools to re-render to be able to do what you need to do to deploy the import you manually modified.

I think it would be sufficient to push the latest version of the import tool to the Galaxy toolshed. Maybe you can do this manually?

[Oddant1]

@bernt-matthias I believe it should be there now via this