Pickling Error

[RSherman15]

I'm coming across the following error when trying to run a basic example (as specified in the documentation) to simulate reads.

Can't pickle <function __rule_5 at 0x7f466e8c4d08>: it's not the same object as snakemake.workflow.__rule_5

I installed using miniconda3 with python 3.5.6. on Linux. I noticed perhaps this is a python 3.5 issue with Snakemake, but not being familiar with Snakemake, this thread did not elucidate how to fix this issue other than downgrading my python version: .

I attempted to start a new conda environment with python 3.4 and rnftools, and instead run into a pysam/htslib issue:

ImportError in line 1 of /ccb/salz8-1/rsherman/alignmentSimulation/rnfToolsScripts/Snakefile:
libhts.so.1: cannot open shared object file: No such file or directory
  File "/ccb/salz8-1/rsherman/alignmentSimulation/rnfToolsScripts/Snakefile", line 1, in <module>
  File "/home/rsherman/bin/miniconda3/envs/rnftools_py34/lib/python3.4/site-packages/rnftools/__init__.py", line 8, in <module>
  File "/home/rsherman/bin/miniconda3/envs/rnftools_py34/lib/python3.4/site-packages/rnftools/mishmash/__init__.py", line 3, in <module>
  File "/home/rsherman/bin/miniconda3/envs/rnftools_py34/lib/python3.4/site-packages/rnftools/mishmash/Source.py", line 6, in <module>
  File "/home/rsherman/bin/miniconda3/envs/rnftools_py34/lib/python3.4/site-packages/pysam/__init__.py", line 5, in <module>

Any aid in getting this running with any version of python would be appreciated.

[RSherman15]

Using python 3.4 and then "conda update pysam" which downgraded samtools, bcftools, and htslib seems to have done the trick.

It does seem though that if python 3.5 is not supported this should be mentioned in the documentation, which specifies python 3.3+ as the requirement.

[karel-brinda]

Hi Rachel,

First of all, thank you very much for using RNFtools and for writing the report.

It looks like the problem does not come directly from RNFtools or Python 3.4/3.5, but from some of the dependencies. I can really imagine how frustrating this can be.

Can't pickle <function __rule_5 at 0x7f466e8c4d08>: it's not the same object as snakemake.workflow.__rule_5

This looks like a bug in Snakemake; I remember similar issues in versions from ~2 years ago.

ImportError in line 1 of /ccb/salz8-1/rsherman/alignmentSimulation/rnfToolsScripts/Snakefile: libhts.so.1: cannot open shared object file: No such file or directory

This looks like a bug in HTSlib/PySam or their Conda packages.

Both Snakemake and PySam are RNFtools dependencies, and both of them had a few releases that were quite buggy. Could you please figure out their versions? I will then improve the specification of RNFtools dependencies in Bioconda.

The best source of information would be the output of conda env export -n rnftools_py34. Could you please upload it here? I haven't managed to reproduce the error on any of my computers so it would be really helpful.

Btw. Please say hello from me to Florian :).

[RSherman15]

So, as I mentioned, using my environment with python 3.4, upgrading pysam changed the versions of samtools, bcftools, and htslib to earlier versions and now it works fine. I'm not sure what they were before, but I know all the versions were newer version numbers. So the output of conda env export -n rnftools_py34 (for those packages, which are the relevant ones) is now working with:

  - bcftools=1.4.1=0
  - htslib=1.4.1=0
  - pysam=0.11.1=py34_0
  - rnftools=0.3.0.2=py34_1
  - samtools=1.4.1=0

Where as for my other env (with python 3.5), which is not working, the versions of those are:

  - bcftools=1.9=h4da6232_0
  - htslib=1.7=0
  - pysam=0.14.1=py35hae42fb6_1
  - rnftools=0.3.1.2=py35_2
  - samtools=1.7=1

I just ran conda update pysam in the 3.5 env and that is NOT a fix in 3.5. Where as in 3.4 it downgraded the versions, here it upgrades, and I still get the same error running snakemake. Here are the versions it upgraded to (which also do not work):

  environment location: /home/rsherman/bin/miniconda3/envs/rnftools

  added / updated specs: 
    - pysam

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pysam-0.15.0.1             |   py35hdfb72b2_1         7.8 MB  bioconda
    samtools-1.9               |       h8ee4bcc_1         483 KB  bioconda
    htslib-1.9                 |       hc238db4_4         1.2 MB  bioconda
    ------------------------------------------------------------
                                           Total:         9.4 MB

The following packages will be UPDATED:

    htslib:   1.7-0                 bioconda --> 1.9-hc238db4_4          bioconda
    pysam:    0.14.1-py35hae42fb6_1 bioconda --> 0.15.0.1-py35hdfb72b2_1 bioconda
    samtools: 1.7-1                 bioconda --> 1.9-h8ee4bcc_1          bioconda

[karel-brinda]

Thank you very much. Could you please include also the versions of Snakemake?

[RSherman15]

With my 3.4 environment it's snakemake=3.11.2=py34_1

With 3.5 it's snakemake=3.6.1=py35_0

EDIT: Just ran conda update snakemake in the 3.5 env. This appears to be a fix. Not sure why the original setup of the environment installs an older version of snakemake than the 3.4 env.