search

vpc-ccg / pamir

Discovery and Genotyping of Novel Sequence Insertions in Many Sequenced Individuals
BSD 3-Clause "New" or "Revised" License
8 stars 4 forks source link

Cannot Execute Binary #5

Closed charliecurnin closed 6 years ago

charliecurnin commented 6 years ago

When trying to execute a simple Pamir command,

https://github.com/vpc-ccg/pamir/issues

I get 

=============================================
Project Name      : pamir_test_4
Working Directory : /scratch/reconstructIns/pamir_test_4
=============================================
Creating a new project folder... OK
Checking binary pre-requisites...  [ERROR] File ~/miniconda3/bin/mrsfast cannot be executed. 
FAILED
Traceback (most recent call last):
  File "/home/groups/XXXX/apps/bin/pamir-install/pamir/pamir.py", line 1469, in <module>
    sys.exit(main())
  File "/home/groups/XXXX/apps/bin/pamir-install/pamir/pamir.py", line 1453, in main
    check_binary_preq(config)
  File "/home/groups/XXXX/apps/bin/pamir-install/pamir/pamir.py", line 1165, in check_binary_preq
    logln ("File {0} cannot be executed. ".format(VELVETH) )
UnboundLocalError: local variable 'VELVETH' referenced before assignment

I can, however, execute mrsfast (but not with sh). 

[ccurnin@sh-ln02 login! /scratch/reconstructIns]$ ~/miniconda3/bin/mrsfast --help | head
mrsFAST-Ultra(1)             mrsfast-Ultra Manual             mrsFAST-Ultra(1)

NAME
       mrsfast-ultra

DESCRIPTION
       mrsFAST is a cache oblivious read mapping tool. mrsFAST capable of map-
       ping single and paired end reads to  the  reference  genome.  Bisulfite
[ccurnin@sh-ln02 login! /scratch/reconstructIns]$ sh ~/miniconda3/bin/mrsfast
/home/users/ccurnin/miniconda3/bin/mrsfast: /home/users/ccurnin/miniconda3/bin/mrsfast: cannot execute binary file
charliecurnin commented 6 years ago

I've realized there's an error when I make -j. I don't have sse4, so I modified my Makefile to be 

CC=g++
FLAGS= -O3 -std=c++11 
CFLAGS= -c $(FLAGS) -Wfatal-errors
SOURCES=partition.cc pamir.cc assembler.cc genome.cc aligner.cc extractor.cc common.cc bam_parser.cc sam_parser.cc record.cc sort.cc logger.cc
LDFLAGS=-lm -lz
OBJECTS=$(SOURCES:.cc=.o) 
EXECUTABLE=pamir
all: snp rc es sm mf cm
basic: snp rc es mf cm

rc: 
        g++ -O3 -o recalibrate recalibrate.cc
cm: 
        g++ -O3 -o cleanmega clean_megablast.cc
es: 
        g++ -O3 -o extract_support extract_support.cc common.cc
sm:
        g++ -O3 -o smoother -g -std=c++1y -Wfatal-errors smoother.cc
mf: 
        make with-sse4=no -C ../mrsfast

snp: $(SOURCES) $(EXECUTABLE)

$(EXECUTABLE): $(OBJECTS) 
        $(CC) $(OBJECTS) $(LDFLAGS) -o $@

.cc.o:
        $(CC) $(CFLAGS) $< -o $@ 

clean:
        rm -f *.o pamir recalibrate extract_support smoother cleanmega

Then when I make -j I get

g++ -c -O3 -std=c++11  -Wfatal-errors partition.cc -o partition.o 
g++ -c -O3 -std=c++11  -Wfatal-errors pamir.cc -o pamir.o 
g++ -c -O3 -std=c++11  -Wfatal-errors assembler.cc -o assembler.o 
g++ -c -O3 -std=c++11  -Wfatal-errors genome.cc -o genome.o 
g++ -c -O3 -std=c++11  -Wfatal-errors aligner.cc -o aligner.o 
g++ -c -O3 -std=c++11  -Wfatal-errors extractor.cc -o extractor.o 
g++ -c -O3 -std=c++11  -Wfatal-errors common.cc -o common.o 
g++ -c -O3 -std=c++11  -Wfatal-errors bam_parser.cc -o bam_parser.o 
g++ -c -O3 -std=c++11  -Wfatal-errors sam_parser.cc -o sam_parser.o 
g++ -c -O3 -std=c++11  -Wfatal-errors record.cc -o record.o 
g++ -c -O3 -std=c++11  -Wfatal-errors sort.cc -o sort.o 
g++ -c -O3 -std=c++11  -Wfatal-errors logger.cc -o logger.o 
g++ -O3 -o recalibrate recalibrate.cc
g++ -O3 -o extract_support extract_support.cc common.cc
g++ -O3 -o smoother -g -std=c++1y -Wfatal-errors smoother.cc
make with-sse4=no -C ../mrsfast
g++ -O3 -o cleanmega clean_megablast.cc
make: *** ../mrsfast: No such file or directory.  Stop.
make: *** [mf] Error 2
make: *** Waiting for unfinished jobs....
yenyilin commented 6 years ago

Hi charliecurnin:

I see make -C ../mrsfast instead of make -C mrsfast. Did you manually adjust the location of mrsfast? We might need to figure out why "../mrsfast: No such file or directory" before moving forward.

Thank you! Yen-Yi

charliecurnin commented 6 years ago

Hi,

Thanks for the quick response. I have -C ../mrsfast just because that's what's written in the troubleshooting section of the readme. It should be -C mrsfast instead?

yenyilin commented 6 years ago

Aha, thanks for catching this error.

Please modify it as

make with-sse4=no -C mrsfast

for now. I will modify the readme file later.

Yen-Yi

charliecurnin commented 6 years ago

OK, changing it to -C mrsfast, I can now make -j with no errors

[ccurnin@sh-ln03 login! /share/pamir]$ make -j 
g++ -c -O3 -std=c++11  -Wfatal-errors partition.cc -o partition.o 
g++ -c -O3 -std=c++11  -Wfatal-errors pamir.cc -o pamir.o 
g++ -c -O3 -std=c++11  -Wfatal-errors assembler.cc -o assembler.o 
g++ -c -O3 -std=c++11  -Wfatal-errors genome.cc -o genome.o 
g++ -c -O3 -std=c++11  -Wfatal-errors aligner.cc -o aligner.o 
g++ -c -O3 -std=c++11  -Wfatal-errors extractor.cc -o extractor.o 
g++ -c -O3 -std=c++11  -Wfatal-errors common.cc -o common.o 
g++ -c -O3 -std=c++11  -Wfatal-errors bam_parser.cc -o bam_parser.o 
g++ -c -O3 -std=c++11  -Wfatal-errors sam_parser.cc -o sam_parser.o 
g++ -c -O3 -std=c++11  -Wfatal-errors record.cc -o record.o 
g++ -c -O3 -std=c++11  -Wfatal-errors sort.cc -o sort.o 
g++ -c -O3 -std=c++11  -Wfatal-errors logger.cc -o logger.o 
g++ -O3 -o recalibrate recalibrate.cc
g++ -O3 -o extract_support extract_support.cc common.cc
g++ -O3 -o smoother -g -std=c++1y -Wfatal-errors smoother.cc
make with-sse4=no -C mrsfast
g++ -O3 -o cleanmega clean_megablast.cc
make[1]: Entering directory `/home/groups/XXXX/apps/bin/pamir-installs/8/pamir/mrsfast'
DSSE4=0
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o baseFAST.o baseFAST.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o Sort.o Sort.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o MrsFAST.o MrsFAST.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o Common.o Common.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o CommandLineParser.o CommandLineParser.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o RefGenome.o RefGenome.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o HashTable.o HashTable.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o Reads.o Reads.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o Output.o Output.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o SNPReader.o SNPReader.c
cc -fno-pic -DMRSFAST_VERSION=\""3.4.0"\" -DBUILD_DATE=\""Thu May  3 14:23:20 PDT 2018"\" -O2   -c -o SNPIndexer.o SNPIndexer.c
gcc SNPIndexer.o -o snp_indexer  -lz -lm -pthread -lpthread
gcc -w baseFAST.o Sort.o MrsFAST.o Common.o CommandLineParser.o RefGenome.o HashTable.o Reads.o Output.o SNPReader.o  HELP.o -o mrsfast  -lz -lm -pthread -lpthread
make[1]: Leaving directory `/home/groups/XXXX/apps/bin/pamir-installs/8/pamir/mrsfast'
g++ partition.o pamir.o assembler.o genome.o aligner.o extractor.o common.o bam_parser.o sam_parser.o record.o sort.o logger.o  -lm -lz -o pamir
[ccurnin@sh-ln03 login! /share/pamir]$

Still, when I try to run pamir.py, I get 

=============================================
Project Name      : pamir_test_6
Working Directory : /scratch/reconstructIns/pamir_test_6
=============================================
Creating a new project folder... OK
Checking binary pre-requisites...  [ERROR] File ~/miniconda3/bin/mrsfast cannot be executed. 
FAILED
Traceback (most recent call last):
  File "/home/groups/XXXX/apps/bin/pamir-installs/8/pamir/pamir.py", line 1469, in <module>
    sys.exit(main())
  File "/home/groups/XXXX/apps/bin/pamir-installs/8/pamir/pamir.py", line 1453, in main
    check_binary_preq(config)
  File "/home/groups/XXXX/apps/bin/pamir-installs/8/pamir/pamir.py", line 1165, in check_binary_preq
    logln ("File {0} cannot be executed. ".format(VELVETH) )
UnboundLocalError: local variable 'VELVETH' referenced before assignment
yenyilin commented 6 years ago

Do you mind sharing your pamir.config with me? I am not sure why mrsfast is from miniconda and VELVETH is not given a value.

Thanks! Yen-Yi

charliecurnin commented 6 years ago

Sure. Should mrsfast not be assigned a path? I know it wasn't in the readme, but the template pamir.config file in the repo includes it. 

### UPDATE THE BINARY and DATABASE DIRECTORIES BEFORE RUNNING PAMIR   
### DO NOT INCLUDE ANY SPACE (" ") IN ANY LINE
#########################################################

MRSFAST=~/miniconda3/bin/mrsfast
SAMTOOLS=~/miniconda3/bin/samtools
VELVETH=~/miniconda3/bin/velveth
VELVETG=~/miniconda3/bin/velvetg
BLASTN=~/miniconda3/bin/blastn
BLASTDB=/scratch/XXXX/blast/db/
#########################################################
yenyilin commented 6 years ago

Can you show me the result of ls -la ~/miniconda3/bin/mrsfast

Thanks!

charliecurnin commented 6 years ago

Sure. It's a single file: 

[ccurnin@sh-ln02 login! ~]$ ls -la ~/miniconda3/bin/mrsfast
-rwxrwxr-x 2 ccurnin XXXX 144745 May 22  2017 /home/users/ccurnin/miniconda3/bin/mrsfast
yenyilin commented 6 years ago

I will install miniconda3 to see what did we miss in checking executables. In the meantime do you mind try a new run after you put SAMTOOLS and BLASTDB as the first line in pamir.config instead of MRSFAST? Thank you for your feedbacks!

charliecurnin commented 6 years ago

Sure. Here's the error I'm getting now: 

=============================================
Project Name      : pamir_test_10
Working Directory : /scratch/reconstructIns/pamir_test_10
=============================================
Creating a new project folder... OK
Checking binary pre-requisites...  [ERROR] File ~/miniconda3/bin/samtools cannot be executed. 
FAILED
File ~/miniconda3/bin/samtools cannot be executed.

Here's my pamir.config:

### UPDATE THE BINARY and DATABASE DIRECTORIES BEFORE RUNNING PAMIR   
### DO NOT INCLUDE ANY SPACE (" ") IN ANY LINE
#########################################################

SAMTOOLS=~/miniconda3/bin/samtools
VELVETH=~/miniconda3/bin/velveth
VELVETG=~/miniconda3/bin/velvetg
BLASTN=~/miniconda3/bin/blastn
BLASTDB=/scratch/XXXX/blast/db/
#########################################################
charliecurnin commented 6 years ago

Interesting! Replacing ~ with /home/users/ccurnin/ in my config file seems to be working!

charliecurnin commented 6 years ago

However, I am now getting

=============================================
Project Name      : pamir_test_13
Working Directory : /scratch/reconstructIns/pamir_test_13
=============================================
Checking the project pre-requisites...  FAILED
Pamir can not overwrite an existing project. Please add --resume or change project name.
[ccurnin@sh-ln03 login! /scratch/reconstructIns]$ cat slurm-16219603.out 
=============================================
Project Name      : pamir_test_14
Working Directory : /scratch/reconstructIns/pamir_test_14
=============================================
Creating a new project folder... OK
Checking binary pre-requisites...  ... OK
Sorting bam file... FAILED
/home/users/ccurnin/miniconda3/bin/samtools sort -n -@ 1 -m 10G /scratch/reconstructIns/pamir_test_14/indels.svs.bam.sort.bam /scratch/reconstructIns/pamir_test_14/indels.svs.bam.sort.bam.sorted failed with exit status 1 and message

I'm not sure this is a valid samtools command? When I try to run , I get [bam_sort] Use -T PREFIX / -o FILE to specify temporary and final output files.

This is samtools 1.7

samtools 1.7
Using htslib 1.7
Copyright (C) 2018 Genome Research Ltd.
charliecurnin commented 6 years ago

I've created a new issue for this problem with samtools.