Bioconda CWL test is actually broken because Galaxy fix isn't released yet

[adamnovak]

On second look, I think our integration with Bioconda has actually broken. We seem to use an --experimental-solver option that has been removed. We need to adapt to the change, or force a version of Conda that has the option.

I think we need galaxy-util and galaxy-tool-util >23.0.5, with this PR which is merged but hasn't been released yet.

@mr-c I think this all comes in through cwltool; can you make the Galaxy release happen somehow?

________________________ CWLWorkflowTest.test_bioconda _________________________
[gw2] linux -- Python 3.9.16 /builds/databiosphere/toil/venv/bin/python
Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.9/unittest/case.py", line 592, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/builds/databiosphere/toil/src/toil/test/cwl/cwlTest.py", line 419, in test_bioconda
    self._tester(
  File "/builds/databiosphere/toil/src/toil/test/cwl/cwlTest.py", line 236, in _tester
    cwltoil.main(main_args, stdout=st)
  File "/builds/databiosphere/toil/src/toil/cwl/cwltoil.py", line 3903, in main
    outobj = toil.start(wf1)
  File "/builds/databiosphere/toil/src/toil/common.py", line 1410, in start
    return self._runMainLoop(rootJobDescription)
  File "/builds/databiosphere/toil/src/toil/common.py", line 1892, in _runMainLoop
    return Leader(config=self.config,
  File "/builds/databiosphere/toil/src/toil/leader.py", line 295, in run
    raise FailedJobsException(self.jobStore, failed_jobs, exit_code=self.recommended_fail_exit_code)
toil.exceptions.FailedJobsException: The job store '/tmp/tmp5j7dqpuj' contains 1 failed jobs: 'CWLJob' seqtk_seq kind-CWLJob/instance-ejcl145n v6
Log from job "'CWLJob' seqtk_seq kind-CWLJob/instance-ejcl145n v6" follows:
=========>
    [2023-10-19T18:45:55+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
    [2023-10-19T18:45:55+0000] [MainThread] [I] [toil] Running Toil version 5.13.0a1-c1361d43602e3276ef4ddec32c162dbe15d0a526 on host runner-wpkqx-uz-project-3-concurrent-3.
    [2023-10-19T18:45:55+0000] [MainThread] [I] [toil.worker] Working on job 'CWLJob' seqtk_seq kind-CWLJob/instance-ejcl145n v4
    [2023-10-19T18:45:56+0000] [MainThread] [I] [toil.worker] Loaded body Job('CWLJob' seqtk_seq kind-CWLJob/instance-ejcl145n v4) from description 'CWLJob' seqtk_seq kind-CWLJob/instance-ejcl145n v4
    [2023-10-19T18:45:57+0000] [MainThread] [I] [toil.cwl.cwltoil] Initial work dir: {'class': 'Directory', 'listing': [], 'basename': ''}
    [2023-10-19T18:45:57+0000] [MainThread] [I] [cwltool] [job seqtk_seq] /tmp/aebc79ac521a5bee83a3b716aba3da2b/c0b3/8b28/tmpgp8o8imc/tmp-out3xv3e0cs$ seqtk \
        seq \
        -a \
        /tmp/aebc79ac521a5bee83a3b716aba3da2b/c0b3/8b28/tmpr7w2z4gjll9nmsub/stg1660aa86-999e-43be-97a6-e2a9558b539b/2.fastq > /tmp/aebc79ac521a5bee83a3b716aba3da2b/c0b3/8b28/tmpgp8o8imc/tmp-out3xv3e0cs/out
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk=r93
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk=r93
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk=r93
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk=r93
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk
    usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
    conda: error: unrecognized arguments: --experimental-solver seqtk
    Traceback (most recent call last):
      File "/tmp/aebc79ac521a5bee83a3b716aba3da2b/c0b3/8b28/tmpgp8o8imc/tmp-outx1wthpty/run_job.py", line 99, in <module>
        sys.exit(main(sys.argv))
      File "/tmp/aebc79ac521a5bee83a3b716aba3da2b/c0b3/8b28/tmpgp8o8imc/tmp-outx1wthpty/run_job.py", line 76, in main
        sp = subprocess.Popen(  # nosec
      File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'seqtk'
    [2023-10-19T18:46:07+0000] [MainThread] [W] [cwltool] [job seqtk_seq] exited with status: 1
    [2023-10-19T18:46:07+0000] [MainThread] [W] [cwltool] [job seqtk_seq] completed permanentFail
    [2023-10-19T18:46:07+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
    [2023-10-19T18:46:07+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-6047575663cd401fbac041ba5f0b909b/2.fastq' to path '/tmp/aebc79ac521a5bee83a3b716aba3da2b/c0b3/8b28/tmpjodgpl0f.tmp'
    Traceback (most recent call last):
      File "/builds/databiosphere/toil/src/toil/worker.py", line 408, in workerScript
        job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/builds/databiosphere/toil/src/toil/job.py", line 2828, in _runner
        returnValues = self._run(jobGraph=None, fileStore=fileStore)
      File "/builds/databiosphere/toil/src/toil/job.py", line 2745, in _run
        return self.run(fileStore)
      File "/builds/databiosphere/toil/src/toil/cwl/cwltoil.py", line 2415, in run
        raise cwl_utils.errors.WorkflowException(status)
    cwl_utils.errors.WorkflowException: permanentFail
    [2023-10-19T18:46:08+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host runner-wpkqx-uz-project-3-concurrent-3
<=========

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1425

[mr-c]

@adamnovak I pinged them; in the mean time I'm downgrading our version of galaxy-tool-util and once that is merged I'll make a new cwltool release: https://github.com/common-workflow-language/cwltool/pull/1930

[mr-c]

@adamnovak https://github.com/common-workflow-language/cwltool/releases/tag/3.1.20231020113452 is out; I'll send a PR to here