Errors with Toil v. 3.14.0 installed through Bioconda and v. 3.18.0 installed through pip

[amizeranschi]

Hello,

I'm getting errors with Toil installed through Bioconda in a fresh environment:

conda_path=/export/home/ncit/external/a.mizeranschi/toil_conda
cd ~
wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
bash Miniconda2-latest-Linux-x86_64.sh -b -p ${conda_path}
ln -s ${conda_path}/bin/conda ${conda_path}/bin/toil_conda
export PATH=/export/home/ncit/external/a.mizeranschi/toil_conda/bin:$PATH
toil_conda install --yes -c conda-forge -c bioconda toil

This is what happens if I try to run toil:

$ toil --version
3.14.0

$ toil
Traceback (most recent call last):
  File "/export/home/ncit/external/a.mizeranschi/assemb_conda/bin/toil", line 11, in <module>
    load_entry_point('toil==3.14.0', 'console_scripts', 'toil')()
  File "/export/home/ncit/external/a.mizeranschi/assemb_conda/lib/python2.7/site-packages/toil/utils/toilMain.py", line 16, in main
    printHelp(modules)
  File "/export/home/ncit/external/a.mizeranschi/assemb_conda/lib/python2.7/site-packages/toil/utils/toilMain.py", line 62, in printHelp
    descriptions='\n'.join("%s - %s" % (n, m.__doc__.strip()) for n, m in iteritems(modules))))
  File "/export/home/ncit/external/a.mizeranschi/assemb_conda/lib/python2.7/site-packages/toil/utils/toilMain.py", line 62, in <genexpr>
    descriptions='\n'.join("%s - %s" % (n, m.__doc__.strip()) for n, m in iteritems(modules))))
AttributeError: 'NoneType' object has no attribute 'strip'

while cwltoil doesn't throw any error:

$ cwltoil
usage: cwltoil [-h] [--logOff] [--logCritical] [--logError] [--logWarning]
               [--logInfo] [--logDebug] [--logLevel LOGLEVEL]
               [--logFile LOGFILE] [--rotatingLogging] [--workDir WORKDIR]
               [--stats] [--clean {always,onError,never,onSuccess}]
               [--cleanWorkDir {always,never,onSuccess,onError}]
               [--clusterStats [CLUSTERSTATS]] [--restart]
               [--batchSystem BATCHSYSTEM] [--disableHotDeployment]
[...]

If I then do pip install --upgrade toil, I end up with v. 3.18.0 and the toil command doesn't through any error:

$ toil --version
3.18.0

$ toil

Usage: toil COMMAND ...
       toil --help
       toil COMMAND --help

where COMMAND is one of the following:

status - Tool for reporting on job status.
debug-job - Debug tool for running a toil job locally.
rsync-cluster - Rsyncs into the toil appliance container running on the leader of the cluster
stats - Reports statistical data about a given Toil workflow.
launch-cluster - Launches a toil leader instance with the specified provisioner
destroy-cluster - Terminates the specified cluster and associated resources
debug-file - Debug tool for copying files contained in a toil jobStore.
kill - Kills rogue toil processes.
clean - Delete the job store used by a previous Toil workflow invocation
ssh-cluster - SSHs into the toil appliance container running on the leader of the cluster

However, now the cwltoil command errors out:

$ cwltoil
Traceback (most recent call last):
  File "/export/home/ncit/external/a.mizeranschi/toil_conda/bin/cwltoil", line 7, in <module>
    from toil.cwl.cwltoil import main
  File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 52, in <module>
    import cwltool.command_line_tool
ImportError: No module named command_line_tool

How can I get both toil and cwltoil to work?

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-50

[chapmanb]

Thanks for the report. The underlying cwltool library Toil depends on underwent a refactoring and I don't believe the latest release is fully compatible with it yet. I know @mr-c mentioned he was working on it, but I don't know about the status.

My imperfect advice right now is to use the older Toil if you need CWL support and the newer one if you want to run without CWL. Hope this helps.

[amizeranschi]

OK, thanks a lot for your input. My goal is to run the newer version of Cactus (https://github.com/ComparativeGenomicsToolkit/cactus), so I'm guessing that CWL support won't be needed for that.

Here's hoping that the error with the latest version will be fixed.

[adamnovak]

Should we release a patch for 3.14? Or can we update what Bioconda ships? It's not good for the Toil command to be broken right out of the box.

@amizeranschi It looks like when you try to run cwltoil from your Pip-installed Toil 3.18, you're actually running cwltoil as installed by Conda (/export/home/ncit/external/a.mizeranschi/toil_conda/bin/cwltoil). I think that if you don't install Toil with the CWL extra (pip install --upgrade toil[cwl] or pip install --upgrade toil[all]), there won't even be a cwltoil command provided by the resulting installation.

[amizeranschi]

@adamnovak You were right, the pip command that I ran did not update cwltoil. Thanks for looking into this.

Running pip install --upgrade toil[all] got it to work for Toil v. 3.18. Will this version become available through conda as well?

[adamnovak]

It looks like @chapmanb may have been handling Toil packaging for conda.

We might just need to make a PR against https://github.com/bioconda/bioconda-recipes/blob/master/recipes/toil/meta.yaml to update it. That probably needs to be part of our release process if we want conda to be up to date.

[amizeranschi]

Could Toil also be made compatible with the latest version of cwltool? This would enable it to be installed once again as part of the base Bcbio-nextgen install. See: https://github.com/bcbio/bcbio-nextgen/issues/2710#issuecomment-469088525.

[DailyDreaming]

Stale. Please reopen if this is still an issue.

[amizeranschi]

@DailyDreaming Not sure why this was closed (as it was never addressed in any way), but I can't reopen the issue myself.

I've updated the old thread at https://github.com/bcbio/bcbio-nextgen/issues/2799 but I'm not sure if the problem that @chapmanb mentioned was fixed, i.e. regarding the cwltool dependency. Do you have any idea about this?

[adamnovak]

It's weird you can't reopen it as the submitter. Maybe we have a project setting mis-set.

I think we've since bumped our cwltool dependency a few times; it may now be new enough for the other package that was conflicting with us.

We no longer want people to use cwltoil, though; I think we now call it toil-cwl-runner.

Given that installing Toil 3.18 through Pip entirely, and not mixing it with a Conda install of 3.14, produced both a working toil and cwltoil (now toil-cwl-runner), what exactly is this issue meant to be tracking now? Is the outstanding problem that toil didn't run with no arguments when Bioconda installed Toil 3.14? Is that still a problem with the current Toil shipped in Bioconda?

Or is this issue meant to track the same problem as https://github.com/bcbio/bcbio-nextgen/issues/2799: namely getting bcbio to include Toil by default again?

[adamnovak]

@DailyDreaming It looks like because of https://github.com/isaacs/github/issues/583 nobody can reopen issues if we close them. I guiess we either have to stop telling people to do that, or replace Github, because Github isn't fixing it.

[amizeranschi]

@adamnovak thanks for looking into this. Yes, I was only hoping to have Toil included by default in bcbio's main conda environment, as mentioned in https://github.com/bcbio/bcbio-nextgen/issues/2799, and the last thing mentioned there was about the cwltool problem. If this is now fixed, then hopefully they'll be willing to re-enable Toil.

Come to think of it, seeing as this is just a bcbio issue after all, it might be best to close this issue altogether.