SciLifeLab / facs

Fast and Accurate Classification of Sequences using Bloom filters
http://facs.scilifelab.se/
Other
16 stars 9 forks source link

Add support for memory benchmarking #113

Closed guillermo-carrasco closed 10 years ago

guillermo-carrasco commented 10 years ago

Profiles the memory usage and adds the corresponding information to the final json records.

Check this doc as an example :-)

Important note: Requires the installation of the latest version of memory_profiler, download and install from github.

brainstorm commented 10 years ago

Beautiful, @guillermo-carrasco :D

Should I assume the units reported with the link above are in MiB?

EDIT: Yes, wouldn't make sense otherwise... thanks!

guillermo-carrasco commented 10 years ago

Oh that's what I forgot :-P Yep, it is in MiB :+1:

brainstorm commented 10 years ago

Indeed, there is a problem with memory_profiler with psutil, after the backtrace, while running the fastq_screen testsuite:

Reading configuration from '/pica/h1/roman/dev/facs/tests/data/bin/fastq_screen.conf'
Using 'bowtie' as bowtie path
Using 1 threads for searches
Adding database eschColi_K12
Processing /pica/h1/roman/dev/facs/tests/data/synthetic_fastq/simngs_eschColi_K12_1000000.fastq
Counting sequences in /pica/h1/roman/dev/facs/tests/data/synthetic_fastq/simngs_eschColi_K12_1000000.fastq
Searching /pica/h1/roman/dev/facs/tests/data/synthetic_fastq/simngs_eschColi_K12_1000000.fastq against eschColi_K12
Processing complete
Process MemTimer-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/pica/h1/roman/.virtualenvs/facs/lib/python2.6/site-packages/memory_profiler.py", line 123, in run
    include_children=self.include_children)
  File "/pica/h1/roman/.virtualenvs/facs/lib/python2.6/site-packages/memory_profiler.py", line 51, in _get_memory
    mem += p.get_memory_info()[0] / _TWO_20
  File "/pica/h1/roman/.virtualenvs/facs/lib/python2.6/site-packages/psutil/__init__.py", line 758, in get_memory_info
    return self._platform_impl.get_memory_info()
  File "/pica/h1/roman/.virtualenvs/facs/lib/python2.6/site-packages/psutil/_pslinux.py", line 470, in wrapper
    raise NoSuchProcess(self.pid, self._process_name)
NoSuchProcess: process no longer exists (pid=2092)

The whole testsuite just hangs in there, from the parent process:

$ strace -p 1582
Process 1582 attached - interrupt to quit
read(6,