kmc std::bad_alloc

[ChrisHIV]

Running iva --test TestIVA inside a virtualbox running Ubuntu 16.04 I get the following error:

Copied input test files into here: /home/user/TestIVA
Current working directory: /home/user/TestIVA
Running iva on the test data with the command:
/home/user/.local/bin/iva --threads 1 --pcr_primers hiv_pcr_primers.fa -f reads_1.fq.gz -r reads_2.fq.gz iva.out
The following command failed with exit code 1
/home/user/.local/bin/iva --threads 1 --pcr_primers hiv_pcr_primers.fa -f reads_1.fq.gz -r reads_2.fq.gz iva.out

The output was:

The following command failed with exit code 134
bash run_kmc.sh

The output was:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
run_kmc.sh: line 2: 16564 Aborted                 (core dumped) kmc -fa -m4 -k95 -sf1 -ci25 -cs1000000 -cx1000000 /home/user/TestIVA/iva.out/tmp.common_kmers.ile75uwy/reads.fa kmc_out $PWD > /dev/null

The installation instructions involve

wget http://sun.aei.polsl.pl/kmc/download/kmc_dump

which pulls version 2.3; via http://sun.aei.polsl.pl/REFRESH/index.php?page=projects&project=kmc&subpage=download version 3.0 is available. Using that, same problem. Not sure if this is a kmc bug or an IVA bug; please advise. Cheers

[martinghunt]

That looks like a kmc issue. If that tmp directory is still there you could double check by running kmc -fa -m4 -k95 -sf1 -ci25 -cs1000000 -cx1000000 /home/user/TestIVA/iva.out/tmp.common_kmers.ile75uwy/reads.fa kmc_out $PWD, but I expect you'll get the same result.

Could also be worth cloning the IVA repo and running python3 -m unittest iva.tests.kcount_test. TestKcount.test_run_kmc and seeing if that throws an error.

[ChrisHIV]

I can't test the former as the --keep_files option is ignored by --test. For the latter:

$ python3 -m unittest iva.tests.kcount_test. TestKcount.test_run_kmc
EE
======================================================================
ERROR:  (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module 'iva.tests.kcount_test' has no attribute ''

======================================================================
ERROR: TestKcount (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: TestKcount
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
ImportError: No module named 'TestKcount'

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (errors=2)

[martinghunt]

Sorry, typo. There was an accidental space in there. Try:

python3 -m unittest iva.tests.kcount_test.TestKcount.test_run_kmc

[ChrisHIV]

.
----------------------------------------------------------------------
Ran 1 test in 0.166s

OK

???

[martinghunt]

I didn't expect that! That suggests kmc is running ok. Bear with me, I'll have to get a VM with Ubuntu 16.04 up and running to test this out some more. I guess you followed this? https://github.com/sanger-pathogens/iva/wiki/Installation-notes-for-Ubuntu

[ChrisHIV]

For the most part, though I installed samtools differently, and I'd installed some other stuff first. I'll go through the full set exactly as written on a clean Ubuntu, and see if I get the same bug.

[martinghunt]

OK, let me know and if you do still get it then I'll take a look.

[ChrisHIV]

Following the instructions verbatim on a clean 16.04.3, iva --test TestIVA gives

Running iva in test mode...
Copied input test files into here: /home/testiva/TestIVA
Current working directory: /home/testiva/TestIVA
Running iva on the test data with the command:
/home/testiva/.local/bin/iva --threads 1 --pcr_primers hiv_pcr_primers.fa -f reads_1.fq.gz -r reads_2.fq.gz iva.out
The following command failed with exit code 1
/home/testiva/.local/bin/iva --threads 1 --pcr_primers hiv_pcr_primers.fa -f reads_1.fq.gz -r reads_2.fq.gz iva.out

The output was:

The following command failed with exit code 134
bash run_kmc.sh

The output was:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
run_kmc.sh: line 2:  5308 Aborted                 (core dumped) kmc -fa -m4 -k95 -sf1 -ci25 -cs1000000 -cx1000000 /home/testiva/TestIVA/iva.out/tmp.common_kmers.hi439w_n/reads.fa kmc_out $PWD > /dev/null

Cloning the git repo and running python3 -m unittest iva.tests.kcount_test.TestKcount.test_run_kmc, I now get an error instead of success (perhaps because the default instructions use kmc 2 instead of 3?):

The following command failed with exit code 134
bash run_kmc.sh

The output was:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
run_kmc.sh: line 2:  7824 Aborted                 (core dumped) kmc -fa -m4 -k10 -sf1 -ci2 -cs4 -cx4 /home/testiva/iva/iva/tests/data/kcount_test.run_kmc.fa kmc_out $PWD > /dev/null

E
======================================================================
ERROR: test_run_kmc (iva.tests.kcount_test.TestKcount)
Test test_run_kmc
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/testiva/iva/iva/common.py", line 35, in syscall
    subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'bash run_kmc.sh' returned non-zero exit status 134

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/testiva/iva/iva/tests/kcount_test.py", line 54, in test_run_kmc
    counts_file = kcount._run_kmc(reads, 'tmp.run_kmc', 10, 2, 4)
  File "/home/testiva/iva/iva/kcount.py", line 113, in _run_kmc
    ran_ok = _run_kmc_with_script('run_kmc.sh', reads, kmer_counts_file, kmer, min_count, max_count, 4, verbose, False, threads=threads)
  File "/home/testiva/iva/iva/kcount.py", line 94, in _run_kmc_with_script
    return common.syscall('bash ' + script, allow_fail=allow_fail)
  File "/home/testiva/iva/iva/common.py", line 44, in syscall
    sys.exit(1)
SystemExit: 1

----------------------------------------------------------------------
Ran 1 test in 0.336s

FAILED (errors=1)

[martinghunt]

I reproduced that error (using vagrant ubuntu/xenial64 VM), with the VM having 2GB of RAM.

Putting the RAM up to 4GB fixed it and iva --test TestIVA ran through ok. How much RAM does your machine have?

For completeness, here's the bash history:

$ history
    1  sudo apt-get update
    2  sudo apt-get install python3-pip zlib1g-dev libncurses5-dev libncursesw5-dev
    3  sudo apt-get install mummer samtools
    4  mkdir $HOME/bin
    5  export PATH=$PATH:$HOME/bin/:$HOME/.local/bin/
    6  cd $HOME/bin
    7  wget http://sun.aei.polsl.pl/kmc/download/kmc
    8  wget http://sun.aei.polsl.pl/kmc/download/kmc_dump
    9  chmod 755 kmc kmc_dump
   10  cd $HOME/bin
   11  wget http://downloads.sourceforge.net/project/smalt/smalt-0.7.6-bin.tar.gz
   12  tar -zxf smalt-0.7.6-bin.tar.gz
   13  cp -s smalt-0.7.6-bin/smalt_x86_64 smalt
   14  pip3 install --user iva
   15  iva --test TestIVA

[ChrisHIV]

Sorry the delay. Yeah that was it - 2GB in the original VM, fixed by increasing it to 4GB. Thanks!