Fails with uninformative error at start of build

[iqbal-lab]

I'm trying to build a plasmodium PRG. The PRG file itself (linear, string) is built from the VCF by the perl script. but then gramtools fails with uninformative message

gramtools build --gram-directory ./gram --vcf prg_construction/data/pf3k_and_DBPMSPS1and2.vcf --reference reference/Pfalciparum.genome.fasta --max-read-length 150 --debug
2018-04-06 21:52:52,913 gramtools    INFO     Start process: build
2018-04-06 21:52:52,914 gramtools    DEBUG    Checking project file structure
2018-04-06 21:52:52,914 gramtools    DEBUG    Executing command:

perl /nfs/research1/zi/mhunt/Cryptic_production/minos_venvs/minos.0.3.0.gts.multi_allele_fix/lib/python3.6/site-packages/gramtools/utils/vcf_to_linear_prg.pl --outfile ./gram/prg --vcf /nfs/research1/zi/projects/gramtools/standard_datasets/pfalciparum/pf3k_release3_cortex_plus_dblmsps/prg_construction/data/pf3k_and_DBPMSPS1and2.vcf --ref /nfs/research1/zi/projects/gramtools/standard_datasets/pfalciparum/pf3k_release3_cortex_plus_dblmsps/reference/Pfalciparum.genome.fasta

2018-04-06 22:05:00,440 gramtools    INFO     stdout:

Finished printing linear PRG. Final number in alphabet is  3216844

2018-04-06 22:05:00,536 gramtools    DEBUG    Finished executing command: 727.621 seconds
2018-04-06 22:05:00,620 gramtools    DEBUG    Executing command:

/nfs/research1/zi/mhunt/Cryptic_production/minos_venvs/minos.0.3.0.gts.multi_allele_fix/lib/python3.6/site-packages/gramtools/bin/gram build --gram ./gram --kmer-size 15 --max-read-size 150 --max-threads 1 --debug

2018-04-06 22:05:00,620 gramtools    DEBUG    Using current working directory:
/nfs/research1/zi/projects/gramtools/standard_datasets/pfalciparum/pf3k_release3_cortex_plus_dblmsps
2018-04-06 22:05:00,644 gramtools    ERROR    Error code != 0
2018-04-06 22:05:00,645 gramtools    DEBUG    Computing sha256 hash of project paths
2018-04-06 22:07:26,083 gramtools    DEBUG    Saving command report:
./gram/build_report.json
2018-04-06 22:07:26,101 gramtools    INFO     End process: build

[iqbal-lab]

I think it must mean there is something off with the input data that is obvious somehow

[iqbal-lab]

Aha - illegal instruction

run build --gram ./gram --kmer-size 15 --max-read-size 150 --max-threads 1 --debug
Starting program: /nfs/research1/zi/mhunt/Cryptic_production/minos_venvs/minos.0.3.0.gts.multi_allele_fix/lib/python3.6/site-packages/gramtools/bin/gram build --gram ./gram --kmer-size 15 --max-read-size 150 --max-threads 1 --debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: File "/nfs/research1/zi/software/lib64/libstdc++.so.6.0.24-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py:/usr/lib/golang/src/pkg/runtime/runtime-gdb.py".
To enable execution of this file add
        add-auto-load-safe-path /nfs/research1/zi/software/lib64/libstdc++.so.6.0.24-gdb.py
line to your configuration file "/homes/zi/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/homes/zi/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"

Program received signal SIGILL, Illegal instruction.
boost::program_options::basic_command_line_parser<char>::basic_command_line_parser (this=0x7fffffffc350, argc=11, argv=0x7fffffffcb18)
    at /tmp/pip-acvlp3w4-build/cmake-build-debug/libgramtools/include/boost/program_options/detail/parsers.hpp:44
44      /tmp/pip-acvlp3w4-build/cmake-build-debug/libgramtools/include/boost/program_options/detail/parsers.hpp: No such file or directory.
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.0.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  boost::program_options::basic_command_line_parser<char>::basic_command_line_parser (this=0x7fffffffc350, argc=11, argv=0x7fffffffcb18)
    at /tmp/pip-acvlp3w4-build/cmake-build-debug/libgramtools/include/boost/program_options/detail/parsers.hpp:44
#1  0x00000000004386d9 in parse_command_line_parameters (argc=argc@entry=11, argv=argv@entry=0x7fffffffcb18)
    at /tmp/pip-acvlp3w4-build/libgramtools/src/main.cpp:62
#2  0x00000000004376b6 in main (argc=11, argv=0x7fffffffcb18) at /tmp/pip-acvlp3w4-build/libgramtools/src/main.cpp:31

[iqbal-lab]

wtf

[iqbal-lab]

valgrind --leak-check=yes /nfs/research1/zi/mhunt/Cryptic_production/minos_venvs/minos.0.3.0.gts.multi_allele_fix/lib/python3.6/site-packages/gramtools/bin/gram build --gram ./gram --kmer-size 15 --max-read-size 150 --max-threads 1 --debug
==96462== Memcheck, a memory error detector.
==96462== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==96462== Using LibVEX rev 1804, a library for dynamic binary translation.
==96462== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==96462== Using valgrind-3.3.0, a dynamic binary instrumentation framework.
==96462== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==96462== For more details, rerun with: -v
==96462== 
--96462-- Warning: DWARF2 CFI reader: unhandled DW_OP_ opcode 0x2a

valgrind: m_debuginfo/readdwarf.c:2152 (copy_convert_CfiExpr_tree): Assertion 'srcix >= 0 && srcix < VG_(sizeXA)(srcxa)' failed.

[ffranr]

@iqbal-lab On which system did you run these commands? Did you use LSF?

[iqbal-lab]

On EBI, once with LSF and also without

[ffranr]

@iqbal-lab OK. What was the bsub command that you executed?

Previously, on slack, you said: "The backend works when I run it directly but via LSF I get that error" Is this the same issue?

Is this the command that you executed? (I made this command up) bsub.py --norun 600 mhc gramtools build --gram-directory ./gram --vcf prg_construction/data/pf3k_and_DBPMSPS1and2.vcf --reference reference/Pfalciparum.genome.fasta --max-read-length 150 --debug

[iqbal-lab]

Any hints what this was about Robyn? Environment somehow?

[iqbal-lab]

This is not a complaint - I know you're doing other stuff - but just recording that I am blocked by this

[ffranr]

@iqbal-lab To move this forward, there are a few things we can try. For instance, simplifying the bsub command: We've seen that gramtools runs correctly when the bsub command is very simple (e.g. bsub.py 2 gram gramtools ...). These options are not necessary: -Is and -E.

Another strategy is to temporarily increase the memory limit for the gramtools VM (gramtools-dev.ebi.ac.uk). Gramtools may run correctly within the VM. We've seen that gramtools runs correctly when the bsub command is not used at all.

[martinghunt]

bsub.py 2 gram gramtools... isn't that simple. It gets converted to this: bsub -E 'test -e /homes/mhunt' -R "select[mem>2000] rusage[mem=2000]" -M2000 -o gram.o -e gram.e -J gram gramtools

[ffranr]

Great! That (probably) narrows down the problem to the interactive shell (-Is) option. Or the project option.

[iqbal-lab]

I get the problem even without the interactive shell , so can't be that.

[iqbal-lab]

Robyn, have you tried precisely the command I ran and reproduced it? Or is it only me?

[iqbal-lab]

Not reproducible on tip opf master

gramtools --version { "version_number": "0.5.0", "last_git_commit_hash": "ff3e2d6f1b138c6839d8db7d83b51df97ef1e3a7", "truncated_git_commits": [ "ff3e2d6 - Robyn Ffrancon, 1525871127 : enhancement: if kmer size 8 or less, generate all possible kmers and build kmer index", "8db0f15 - Robyn Ffrancon, 1525860767 : enhancement: paths no longer memorised during building kmer index; cleanup", "c806726 - Robyn Ffrancon, 1524587729 : added docker config file", "63c175a - Robyn Ffrancon, 1523897681 : fix: per base coverage recorded when allele length greater than read length", "ef120fb - Robyn Ffrancon, 1523868010 : CMake ensures compiler supports C++17" ] }