unable to run executables in /apps

[chouellaine]

Hello! What architecture are folks running the code on? I'm on an Intel Core i9 MacOS machine and get an executable format error for all the executables in the apps directory.

[jinzhuangdou]

We tested the pipeline in linux environment.

[chouellaine]

gotcha, what are the versions of vftools, samtools, bmftools, and htslib used?

[jinzhuangdou]

It is not necessary to prepare for such tools. They are already included in app folder.

[chouellaine]

I unfortunately cannot run those executables with the machine I have and would rather recompile them instead of spinning up a virtual machine. It would be helpful to get the version numbers of those packages!

[jinzhuangdou]

Could you post the error message when you run them? If showing library issues, you can try whether it works that mentioned in FAQ section?

[chouellaine]

[2023-01-24 15:57:11,655] INFO     Monopogen.py Preparing varint calling pipeline...
[2023-01-24 15:57:11,655] INFO     Monopogen.py Parameters in effect:
[2023-01-24 15:57:11,655] INFO     Monopogen.py --subcommand = [SCvarCall]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --bamFile = [/Users/echou/Downloads/Monopogen/example/chr20_2Mb.rh.filter.sort.bam]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --step = [all]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --mode = [single]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --chr = [chr20]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --out = [/Users/echou/Downloads/Monopogen/test/out]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --reference = [../example/chr20_2Mb.hg38.fa]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --imputation_panel = [../example/CCDG_14151_B01_GRM_WGS_2020-08-05_chr20.filtered.shapeit2-duohmm-phased.vcf.gz]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --depth_filter = [200]
[2023-01-24 15:57:11,655] INFO     Monopogen.py --depth_filter_monovar = [50]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --alt_ratio = [0.1]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --wildCluster = [2]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --mutationCluster = [1]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --max_mismatch = [3]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --max_softClipped = [5]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --app_path = [../apps]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --cell_cluster = [../example/cell_cluster.csv]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --logfile = [out_chr20.log]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --samtools = [/Users/echou/Downloads/Monopogen/apps/samtools]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --bcftools = [/Users/echou/Downloads/Monopogen/apps/bcftools]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --bgzip = [/Users/echou/Downloads/Monopogen/apps/bgzip]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --java = [java]
[2023-01-24 15:57:11,656] INFO     Monopogen.py --beagle = [/Users/echou/Downloads/Monopogen/apps/beagle.27Jul16.86a.jar]
[2023-01-24 15:57:11,656] INFO     Monopogen.py Checking existence of essenstial resource files...
[2023-01-24 15:57:11,656] INFO     Monopogen.py Checking dependencies...
[2023-01-24 15:57:11,775] INFO     Monopogen.py Filtering bam files...
sh: /Users/echou/Downloads/Monopogen/apps/samtools: cannot execute binary file
[2023-01-24 15:57:15,739] INFO     Monopogen.py Performing Variant Calling...
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 1: /Users/echou/Downloads/Monopogen/apps/samtools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 1: /Users/echou/Downloads/Monopogen/apps/bcftools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 1: /Users/echou/Downloads/Monopogen/apps/bcftools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 1: /Users/echou/Downloads/Monopogen/apps/bgzip: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 2: /Users/echou/Downloads/Monopogen/apps/bcftools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 2: /Users/echou/Downloads/Monopogen/apps/bcftools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 2: /Users/echou/Downloads/Monopogen/apps/bgzip: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 5: /Users/echou/Downloads/Monopogen/apps/samtools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 5: /Users/echou/Downloads/Monopogen/apps/bcftools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 5: /Users/echou/Downloads/Monopogen/apps/bcftools: cannot execute binary file
/Users/echou/Downloads/Monopogen/test/out/Scriptchr20/runBeagle.sh: line 5: /Users/echou/Downloads/Monopogen/apps/bgzip: cannot execute binary file
[2023-01-24 15:57:16,534] INFO     Monopogen.py Generating variant statistical information ...
[E::hts_open_format] Failed to open file "/Users/echou/Downloads/Monopogen/test/out/SCvarCall/chr20.gp.vcf.gz" : No such file or directory
Traceback (most recent call last):
  File "../src/Monopogen.py", line 519, in <module>
    main()
  File "../src/Monopogen.py", line 514, in main
    args.func(args)
  File "../src/Monopogen.py", line 398, in SCvarCall
    getDPinfo(args)
  File "../src/Monopogen.py", line 152, in getDPinfo
    gp_vcf_in = VariantFile(out + "/SCvarCall/" +  args.chr + ".gp.vcf.gz") 
  File "pysam/libcbcf.pyx", line 4119, in pysam.libcbcf.VariantFile.__init__
  File "pysam/libcbcf.pyx", line 4344, in pysam.libcbcf.VariantFile.open
FileNotFoundError: [Errno 2] could not open variant file `b'/Users/echou/Downloads/Monopogen/test/out/SCvarCall/chr20.gp.vcf.gz'`: No such file or directory

I have done what is mentioned in the FAQ section re: permission errors and path variable.

[samantha-hoffman]

I'm encountering a similar error with running the germline step:

python src/Monopogen.py germline -a apps -t 1 -r test/region.lst -p example/ -g example/chr20_2Mb.hg38.fa -s all -o out
[2023-08-19 17:24:32,135] INFO     Monopogen.py Performing germline variant calling...
[2023-08-19 17:24:32,135] INFO     germline.py Parameters in effect:
[2023-08-19 17:24:32,136] INFO     germline.py --subcommand = [germline]
[2023-08-19 17:24:32,136] INFO     germline.py --region = [test/region.lst]
[2023-08-19 17:24:32,136] INFO     germline.py --step = [all]
[2023-08-19 17:24:32,136] INFO     germline.py --out = [out]
[2023-08-19 17:24:32,136] INFO     germline.py --reference = [example/chr20_2Mb.hg38.fa]
[2023-08-19 17:24:32,136] INFO     germline.py --imputation_panel = [example/]
[2023-08-19 17:24:32,136] INFO     germline.py --max_softClipped = [1]
[2023-08-19 17:24:32,136] INFO     germline.py --app_path = [apps]
[2023-08-19 17:24:32,136] INFO     germline.py --nthreads = [1]
[2023-08-19 17:24:32,136] INFO     germline.py --norun = [FALSE]
[2023-08-19 17:24:32,136] INFO     Monopogen.py Checking existence of essenstial resource files...
[2023-08-19 17:24:32,181] INFO     Monopogen.py Checking dependencies...
['bash out/Script/runGermline_chr20.sh']
[mpileup] 2 samples in 2 input files
(mpileup) Max depth is above 1M. Potential memory hog!
Lines   total/split/realigned/skipped:  1998600/19910/1485/0
java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
gzip: out/germline/chr20.gp.vcf.gz: No such file or directory
out/germline/chr20.gp.vcf.gz: No such file or directory
java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
[2023-08-19 17:28:11,544] INFO     Monopogen.py Success! See instructions above.

any recommendations?

[jinzhuangdou]

It seems you are lacking libjli.so. You can check your system and locate its path then put it in LD_LIBRARY_PATH See FAQs 3 (I have put libbz2.so.1.0 in app folder)

bcftools: error while loading shared libraries: libbz2.so.1.0: not able to open shared object file: No such file or directly Adding the apps folder of Monopogen in your library environment

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/xx/apps

[mestia]

We tested the pipeline in linux environment.

Please remove the complete /apps directory and instead provide the information how to install the tools. This is wrong on so many levels. 1) binary compatibility, one can not run these tools on different linux systems, there are so many! 2) security, one shouldn't run some binary from internet! please provide the source instead or just instructions how to install the tools. 3) what versions of samtools, bcftools and so on do you use? Please provide the versions, otherwise we run into trouble, since the up-to-date samtools doesn't seem to recognize -t flag for example, same might apply to the other tools.

Thank you!

[ayavuzcakir]

Same, I encounter a problem when I run it, it would be better to run tools from a conda environment, or separately. The error I have; **

$ python3 ${path}/src/Monopogen.py preProcess -t 8 -b bam.lst -o out -a ${path}/apps

[2023-08-31 17:51:28,891] INFO Monopogen.py Performing data preprocess before variant calling... [2023-08-31 17:51:28,891] INFO germline.py Parameters in effect: [2023-08-31 17:51:28,891] INFO germline.py --subcommand = [preProcess] [2023-08-31 17:51:28,891] INFO germline.py --bamFile = [/Monopogen/test/bam.lst] [2023-08-31 17:51:28,891] INFO germline.py --out = [out] [2023-08-31 17:51:28,891] INFO germline.py --app_path = [/Monopogen/apps] [2023-08-31 17:51:28,891] INFO germline.py --max_mismatch = [3] [2023-08-31 17:51:28,891] INFO germline.py --nthreads = [8] [2023-08-31 17:51:28,903] DEBUG Monopogen.py PreProcessing sample ID1 /Monopogen/apps/samtools: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory /Monopogen/apps/samtools: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory

**

[jinzhuangdou]

Thanks for your comments. We have included the version of bcftools/samtools/beagle so that you can compile them in your own system. The output format of such tools changed a lot for different versions. This makes us difficult to manage the downstream output.