search

bergmanlab / mcclintock

Meta-pipeline to identify transposable element insertions using next generation sequencing data
93 stars 30 forks source link

Installation of ngs_te_mapper2 fails #92

Closed ValentinaBoP closed 1 year ago

ValentinaBoP commented 2 years ago

Hi!

I'm encountering an error when installing the tools through the python3 mcclintock.py --install. Everything gets installed correctly but ngs_te_mapper2:

INSTALL          Installing scripts for:ngs_te_mapper2
Job counts:
    count   jobs
    1   ngs_te_mapper2
    1
Traceback (most recent call last):
  File "/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/.snakemake/scripts/tmpbl9tp3wa.ngs_te_mapper2.py", line 3, in <module>
    import sys; sys.path.extend(['/proj/sllstore2017073/private/Valentina/Conda/mcclintock/lib/python3.7/site-packages', '/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/scripts']); import pickle; snakemake = pickle.loads(b'\x80\x03csnakemake.script\nSnakemake\nq\x00)\x81q\x01}q\x02(X\x05\x00\x00\x00inputq\x03csnakemake.io\nInputFiles\nq\x04)\x81q\x05}q\x06(X\x06\x00\x00\x00_namesq\x07}q\x08X\x12\x00\x00\x00_allowed_overridesq\t]q\n(X\x05\x00\x00\x00indexq\x0bX\x04\x00\x00\x00sortq\x0ceh\x0bcfunctools\npartial\nq\rcbuiltins\ngetattr\nq\x0ecsnakemake.io\nNamedlist\nq\x0fX\x0f\x00\x00\x00_used_attributeq\x10\x86q\x11Rq\x12\x85q\x13Rq\x14(h\x12)}q\x15X\x05\x00\x00\x00_nameq\x16h\x0bsNtq\x17bh\x0ch\rh\x12\x85q\x18Rq\x19(h\x12)}q\x1ah\x16h\x0csNtq\x1bbubX\x06\x00\x00\x00outputq\x1ccsnakemake.io\nOutputFiles\nq\x1d)\x81q\x1eXw\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/ngs_te_mapper2/sourceCode/ngs_te_mapper2.pyq\x1fa}q (h\x07}q!h\t]q"(h\x0bh\x0ceh\x0bh\rh\x12\x85q#Rq$(h\x12)}q%h\x16h\x0bsNtq&bh\x0ch\rh\x12\x85q\'Rq((h\x12)}q)h\x16h\x0csNtq*bubX\x06\x00\x00\x00paramsq+csnakemake.io\nParams\nq,)\x81q-(Xa\x00\x00\x00https://github.com/bergmanlab/ngs_te_mapper2/archive/220619a3dd00b8c05c3b96328eb2424c6aad623e.zipq.X \x00\x00\x0095198ab68fcb4f9dc267460a7f906faeq/Xm\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/ngs_te_mapper2/ngs_te_mapper2.zipq0XV\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install//log/install.logq1e}q2(h\x07}q3(X\x03\x00\x00\x00urlq4K\x00N\x86q5X\x03\x00\x00\x00md5q6K\x01N\x86q7X\x07\x00\x00\x00zipfileq8K\x02N\x86q9X\x03\x00\x00\x00logq:K\x03N\x86q;uh\t]q<(h\x0bh\x0ceh\x0bh\rh\x12\x85q=Rq>(h\x12)}q?h\x16h\x0bsNtq@bh\x0ch\rh\x12\x85qARqB(h\x12)}qCh\x16h\x0csNtqDbh4h.h6h/h8h0h:h1ubX\t\x00\x00\x00wildcardsqEcsnakemake.io\nWildcards\nqF)\x81qG}qH(h\x07}qIh\t]qJ(h\x0bh\x0ceh\x0bh\rh\x12\x85qKRqL(h\x12)}qMh\x16h\x0bsNtqNbh\x0ch\rh\x12\x85qORqP(h\x12)}qQh\x16h\x0csNtqRbubX\x07\x00\x00\x00threadsqSK\x01X\t\x00\x00\x00resourcesqTcsnakemake.io\nResources\nqU)\x81qV(K\x01K\x01e}qW(h\x07}qX(X\x06\x00\x00\x00_coresqYK\x00N\x86qZX\x06\x00\x00\x00_nodesq[K\x01N\x86q\\uh\t]q](h\x0bh\x0ceh\x0bh\rh\x12\x85q^Rq_(h\x12)}q`h\x16h\x0bsNtqabh\x0ch\rh\x12\x85qbRqc(h\x12)}qdh\x16h\x0csNtqebhYK\x01h[K\x01ubh:csnakemake.io\nLog\nqf)\x81qg}qh(h\x07}qih\t]qj(h\x0bh\x0ceh\x0bh\rh\x12\x85qkRql(h\x12)}qmh\x16h\x0bsNtqnbh\x0ch\rh\x12\x85qoRqp(h\x12)}qqh\x16h\x0csNtqrbubX\x06\x00\x00\x00configqs}qt(X\x05\x00\x00\x00pathsquccollections\nOrderedDict\nqv)Rqw(X\x08\x00\x00\x00mcc_pathqxX=\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintockqyX\x07\x00\x00\x00installqzXF\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/q{X\x07\x00\x00\x00log_dirq|XK\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install//log/q}uX\x04\x00\x00\x00URLsq~hv)Rq\x7f(X\t\x00\x00\x00te-locateq\x80XA\x00\x00\x00https://downloads.sourceforge.net/project/te-locate/TE-locate.tarq\x81X\x08\x00\x00\x00retroseqq\x82XT\x00\x00\x00https://github.com/tk2/RetroSeq/archive/9d4f3b5270af2383f40e6e7ea1204ea718365db2.zipq\x83X\x04\x00\x00\x00tempq\x84X^\x00\x00\x00https://github.com/JialiUMassWengLab/TEMP/archive/4f67e1da836721a9f0999efa52e1e648fedb75fc.zipq\x85X\x05\x00\x00\x00temp2q\x86XV\x00\x00\x00https://github.com/weng-lab/TEMP2/archive/4d25d050dc7832b82374502573e882a242457a0c.zipq\x87X\x08\x00\x00\x00relocateq\x88XW\x00\x00\x00https://github.com/srobb1/RelocaTE/archive/ce3a2066e15f5c14e2887fdf8dce0485e1750e5b.zipq\x89X\r\x00\x00\x00ngs_te_mapperq\x8aX`\x00\x00\x00https://github.com/bergmanlab/ngs_te_mapper/archive/f9f48996ac346ac86d57edbd00534aa1227b753e.zipq\x8bX\x0e\x00\x00\x00ngs_te_mapper2q\x8ch.X\r\x00\x00\x00popoolationteq\x8dXM\x00\x00\x00http://downloads.sourceforge.net/project/popoolationte/popoolationte_1.02.zipq\x8eX\x0e\x00\x00\x00popoolationte2q\x8fXL\x00\x00\x00http://downloads.sourceforge.net/project/popoolation-te2/popte2-v1.10.03.jarq\x90X\t\x00\x00\x00relocate2q\x91X:\x00\x00\x00https://github.com/stajichlab/RelocaTE2/archive/v2.0.1.zipq\x92X\x05\x00\x00\x00tepidq\x93XW\x00\x00\x00https://github.com/ListerLab/TEPID/archive/ad46d65b5c41bf8a9171215d49b3ffaecdceaab0.zipq\x94X\x06\x00\x00\x00teflonq\x95XW\x00\x00\x00https://github.com/jradrion/TEFLoN/archive/3e2d67886b70644fd1f7d79263b3c8dbed639e46.zipq\x96X\t\x00\x00\x00jitterbugq\x97XX\x00\x00\x00https://github.com/elzbth/jitterbug/archive/b6b3f9c7ee4af042d4410137269f174a1399b752.zipq\x98X\x07\x00\x00\x00tebreakq\x99XY\x00\x00\x00https://github.com/adamewing/tebreak/archive/3f00badda822dcb6390bbea5a6a1e233cff3e99c.zipq\x9auX\x04\x00\x00\x00MD5sq\x9bhv)Rq\x9c(h\x80X \x00\x00\x00c28e3dd7be89a8efcfbca8939923787cq\x9dh\x82X \x00\x00\x0054030657dd8476f1a65f5f71e809b4c3q\x9eh\x84X \x00\x00\x00f06e25244e9979510a9743bcd9c841a5q\x9fh\x86X \x00\x00\x0055b6449113e85540f8a39bb02cfec9b0q\xa0h\x88X \x00\x00\x0064cc60f65154368518529fb57becc3d6q\xa1h\x8aX \x00\x00\x00a00e9962ee3daca9fac915c23c6af203q\xa2h\x8ch/h\x8dX \x00\x00\x00315d266bd6da7487c919576dc46b92b4q\xa3h\x8fX \x00\x00\x000c06186f0b1df215949f57a5cbb2d039q\xa4h\x91X \x00\x00\x0063eb7dbc09ffeb4063aef4857d198b06q\xa5h\x93X \x00\x00\x0051f61acf138b07842074f054aba780d5q\xa6h\x95X \x00\x00\x00a29d2e7411fd06778186ba322f51fdc9q\xa7h\x97X \x00\x00\x0005f165fb6a79a7a611d09981a355d27eq\xa8h\x99X \x00\x00\x00ac475d3cc4b0435f0a333c8d86f57d3fq\xa9uX\x04\x00\x00\x00ENVsq\xaahv)Rq\xab(X\x08\x00\x00\x00coverageq\xacX[\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_coverage.ymlq\xadh\x80X[\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_telocate.ymlq\xaeh\x82X[\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_retroseq.ymlq\xafh\x84XW\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_temp.ymlq\xb0h\x86XX\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_temp2.ymlq\xb1h\x88X[\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_relocate.ymlq\xb2h\x8aX`\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_ngs_te_mapper.ymlq\xb3h\x8cXa\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_ngs_te_mapper2.ymlq\xb4h\x8dX`\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_popoolationte.ymlq\xb5h\x8fXa\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_popoolationte2.ymlq\xb6h\x91X\\\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_relocate2.ymlq\xb7h\x93XX\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_tepid.ymlq\xb8h\x95XY\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_teflon.ymlq\xb9h\x97X\\\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_jitterbug.ymlq\xbaX\x0b\x00\x00\x00setup_readsq\xbbX]\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_trimgalore.ymlq\xbcX\n\x00\x00\x00processingq\xbdX\\\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_map_reads.ymlq\xbeX\t\x00\x00\x00map_readsq\xbfX\\\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_map_reads.ymlq\xc0X\n\x00\x00\x00trimgaloreq\xc1X]\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_trimgalore.ymlq\xc2h\x99XZ\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/mcc_tebreak.ymlq\xc3uX\x06\x00\x00\x00outputq\xc4hv)Rq\xc5(h\xacXa\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/coverage/coverage.logq\xc6h\x80Xb\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/te-locate/TE_locate.plq\xc7h\x82Xd\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/retroseq/bin/retroseq.plq\xc8h\x84Xj\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/temp/scripts/TEMP_Insertion.shq\xc9h\x86XW\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/temp2/TEMP2q\xcah\x88Xx\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/relocate/scripts/relocaTE_insertionFinder.plq\xcbh\x8aXy\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/ngs_te_mapper/sourceCode/run_ngs_te_mapper.shq\xcch\x8cXw\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/ngs_te_mapper2/sourceCode/ngs_te_mapper2.pyq\xcdh\x8dXt\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/popoolationte/identify-te-insertsites.plq\xceh\x8fXn\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/popoolationte2/popte2-v1.10.03.jarq\xcfh\x91Xc\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/relocate2/relocaTE2.logq\xd0h\x93X[\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/tepid/tepid-mapq\xd1h\x95Xa\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/teflon/teflon.v0.4.pyq\xd2h\x97Xb\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/jitterbug/jitterbug.pyq\xd3h\xbfXc\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/map_reads/map_reads.logq\xd4h\xc1Xe\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/trimgalore/trimgalore.logq\xd5h\x99Xc\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/tebreak/tebreak/tebreakq\xd6uuX\x04\x00\x00\x00ruleq\xd7X\x0e\x00\x00\x00ngs_te_mapper2q\xd8X\x0f\x00\x00\x00bench_iterationq\xd9NX\t\x00\x00\x00scriptdirq\xdaXM\x00\x00\x00/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/scriptsq\xdbub.'); from snakemake.logging import logger; logger.printshellcmds = False; __real_file__ = __file__; __file__ = '/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/scripts/ngs_te_mapper2.py';
AttributeError: type object 'Namedlist' has no attribute '_used_attribute'
[Mon Mar 21 16:05:46 2022]
Error in rule ngs_te_mapper2:
    jobid: 0
    output: /crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/ngs_te_mapper2/sourceCode/ngs_te_mapper2.py
    conda-env: /crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/conda/cf895219

RuleException:
CalledProcessError in line 116 of /crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/Snakefile:
Command 'source /proj/sllstore2017073/private/Valentina/Conda/mcclintock/bin/activate '/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/envs/conda/cf895219'; set -euo pipefail;  python /crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/.snakemake/scripts/tmpbl9tp3wa.ngs_te_mapper2.py' returned non-zero exit status 1.
  File "/proj/sllstore2017073/private/Valentina/Conda/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2340, in run_wrapper
  File "/crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/Snakefile", line 116, in __rule_ngs_te_mapper2
  File "/proj/sllstore2017073/private/Valentina/Conda/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/proj/sllstore2017073/private/Valentina/Conda/mcclintock/lib/python3.7/concurrent/futures/thread.py", line 57, in run
  File "/proj/sllstore2017073/private/Valentina/Conda/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/proj/sllstore2017073/private/Valentina/Conda/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2352, in run_wrapper
Exiting because a job execution failed. Look above for error message
snakemake --use-conda --conda-frontend=mamba --conda-prefix /crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install//envs/conda --configfile /crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install//config.json --cores 1 --nolock /crex/proj/sllstore2017073/private/Valentina/Tools/mcclintock/install/tools/ngs_te_mapper2/sourceCode/ngs_te_mapper2.py --quiet

I also tried to install ngs_te_mapper2 manually but then it complains that the version installed is the wrong one and I don't manage to find the right one unfortunately.

Thank you for your help! Valentina

cbergman commented 2 years ago

Hi Valentina

Thanks for the feedback. To help us resolve this problem, could you tell us which OS and version you are running on your machine? Also, can you confirm that you used the mamba-based installation instructions detailed here: https://github.com/bergmanlab/mcclintock/#install. Finally can you tell us which revision of the McClintock code you are running (you can do this by typing git log | head -n 1 inside the mcclintock git repository)?

On thing you can try in the mean time is to do a clean install (remove the entire mcclintock git repository and reclone) then try installing just ngs_te_mapper2 using python3 mcclintock.py --install -m ngs_te_mapper2.

Thanks, Casey

ValentinaBoP commented 2 years ago

Hi Casey,

Yes, I followed the installation instructions on the Github page and used mamba. The OS is: CentOS Linux release 7.9.2009 McClintock version: commit 6d9dda21538f9a97795c4e081c7a9a77ac0c4b5f

I also tried what you suggested about the clean install but I got the same exact error.

cbergman commented 2 years ago

Hi @ValentinaBoP

Thanks for the additional information. I tried replicating your installation problem on a fresh install of a CentOS 7.9.2009 virtual machine running on MacOS parallels. Unfortunately, I could not replicate your error and was able to cleanly install ngs_te_mapper2 via McClintock:


(base) [cbergman@localhost mcclintock]$ conda activate mcclintock
(mcclintock) [cbergman@localhost mcclintock]$ python3 mcclintock.py --install -m ngs_te_mapper2
INSTALL          installing dependencies
INSTALL          WARNING: this could take awhile
INSTALL          Installing scripts for:ngs_te_mapper2
Job counts:
    count   jobs
    1   ngs_te_mapper2
    1
downloading  https://github.com/bergmanlab/ngs_te_mapper2/archive/220619a3dd00b8c05c3b96328eb2424c6aad623e.zip to /home/cbergman/mcclintock/install/tools/ngs_te_mapper2/ngs_te_mapper2.zip
download complete
checking md5 of  /home/cbergman/mcclintock/install/tools/ngs_te_mapper2/ngs_te_mapper2.zip
MD5 hash of  /home/cbergman/mcclintock/install/tools/ngs_te_mapper2/ngs_te_mapper2.zip matches expected

(mcclintock) [cbergman@localhost mcclintock]$ more /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

(mcclintock) [cbergman@localhost mcclintock]$ history
    1  wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O $HOME//miniconda.sh
    2  bash ~/miniconda.sh -b -p $HOME/miniconda # silent mode
    3  echo "export PATH=\$PATH:\$HOME/miniconda/bin" >> $HOME/.bashrc # add to .bashrc
    4  source $HOME/.bashrc
    5  conda init
    6  conda update -y conda
    7  conda install -y mamba=0.21.2 -c conda-forge
    8  conda install -y git
    9  git clone https://github.com/bergmanlab/mcclintock.git
   10  cd mcclintock/
   11  mamba env create -f install/envs/mcclintock.yml --name mcclintock
   12  conda activate mcclintock
   13  python3 mcclintock.py --install -m ngs_te_mapper2
   14  more /etc/os-release 
   15  history

I have also had success installing all McClintock components on other versions of CentOS (8.4.2105, 6.10). From the fact that you can install other components, it appears snakemake is working as it should. Thus, I suspect that it is not a CentOS or CentOS version problem, but rather the issue is most likely with your user account or the installation of CentOS on your machine.

If you are trying to install on a workstation, I would suggest trying a clean install in a new user account to see if there is something in your user environment that is causing this issue. If you are trying to install on a central cluster, I would contact a colleague or your system administrator to see if they can try installing on a different user account.

Please let us know what you find.

Best regards, Casey

ValentinaBoP commented 2 years ago

Thank you so much for the detailed help! I'm using a cluster, will ask help to the IT support and share the solution (hopefully) asap

cbergman commented 2 years ago

Hi @ValentinaBoP. Did you make progress resolving this issue? Thanks, Casey.

ValentinaBoP commented 2 years ago

Hi! I haven't managed to solve it. I also tried to install the McClintock on my own Mac laptop and get the same error for ngs_te_mapper2

ValentinaBoP commented 1 year ago

Hi! In the end the IT support team managed to install McClintock and they also had some problems because of a missing dependency regarding compiling Fortran.