search

jdidion / atropos

An NGS read trimming tool that is specific, sensitive, and speedy. (production)
Other
120 stars 15 forks source link

Segmentation Fault After Closing SequenceReader #88

Closed jessrosenfield closed 4 years ago

jessrosenfield commented 4 years ago

I cannot reliably reproduce this error, but I have been getting segmentation faults every now and then on single-end adapter-match and paired-end insert-match trimming. Upon adding faulthandler.enable() to atropos/commands/__init__.py I was able to get this output after a couple reruns of the following command (using a self-built docker image w/ a ubuntu 16.04 base on an ec2 instance with 32 cpus). I hope this information is still useful to have on hand despite the absence of the docker image and input files used (I've left off the -o, -p, -pe1, and -pe2 args).

/usr/local/bin/atropos trim \
        -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCA \
        -A AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT \
        --no-cache-adapters \
        --no-default-adapters \
        -T 32 \
        --aligner insert \
        --preserve-order

The trace indicates the error is occurring here: https://github.com/jdidion/atropos/blob/1.1/atropos/io/seqio.py#L93

        stderr:
            Fatal Python error: Segmentation fault

            Thread 0x00007fd7bb834700 (most recent call first):
              File "/usr/lib/python3.5/threading.py", line 293 in wait
              File "/usr/lib/python3.5/multiprocessing/queues.py", line 229 in _feed
              File "/usr/lib/python3.5/threading.py", line 862 in run
              File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
              File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

            Current thread 0x00007fd7bf81b700 (most recent call first):
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/io/seqio.py", line 93 in close
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/io/seqio.py", line 463 in close
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/base.py", line 358 in finish
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/base.py", line 283 in __next__
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/multicore.py", line 507 in enqueue_all
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/multicore.py", line 291 in __call__
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/util/__init__.py", line 727 in run_interruptible
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/multicore.py", line 267 in run
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/trim/__init__.py", line 719 in run_parallel
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/trim/__init__.py", line 598 in __call__
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/base.py", line 332 in run
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/__init__.py", line 136 in run_command
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/__init__.py", line 71 in execute
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/atropos/commands/__init__.py", line 219 in execute_cli
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/EGG-INFO/scripts/atropos", line 23 in main
              File "/usr/local/lib/python3.5/dist-packages/atropos-1.1.21-py3.5-linux-x86_64.egg/EGG-INFO/scripts/atropos", line 26 in <module>
              File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1504 in run_script
              File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 719 in run_script
              File "/usr/local/bin/atropos", line 4 in <module>
jdidion commented 4 years ago

Thanks for reporting this @jessrosenfield. What version of Atropos are you using?

jessrosenfield commented 4 years ago

I'm using 1.1.21 via https://github.com/jdidion/atropos/archive/1.1.21.tar.gz

On Mon, Dec 23, 2019 at 7:31 PM John Didion notifications@github.com wrote:

Thanks for reporting this @jessrosenfield https://github.com/jessrosenfield. What version of Atropos are you using?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jdidion/atropos/issues/88?email_source=notifications&email_token=AALXJPLMQDZT5QO5TIQREXTQ2F7BNA5CNFSM4J6ZA4M2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHSMLUA#issuecomment-568640976, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALXJPPLGRESXBUYALDEBM3Q2F7BNANCNFSM4J6ZA4MQ .

jdidion commented 4 years ago

Thanks @jessrosenfield. Do you know if it fails on the same line of code every time? That line is simply setting a variable to None and is only ever called by a single thread, so I can't see how it would cause a SegmentationFault.

My suspicion is that this is instead an out-of-memory error. Since you said you are running this in a Docker container, can you check that the Docker engine is configured to allocate a sufficient amount of memory to the container? Try 32 GB - that should be overkill, but once you figure out whether that is the fix you can start scaling back, perhaps using a profiler to determine the peak usage.

jessrosenfield commented 4 years ago

Sorry for the delay, I've been on vacation. This was already run with excessive memory (64GB) allocated on the ec2 instance and the docker process was only using a small fraction of that memory. When I return I will do some profiling to check that the docker container has proper access to the memory on the instance and do additional profiling of memory used over time.

On Wed, Dec 25, 2019, 13:51 John Didion notifications@github.com wrote:

Thanks @jessrosenfield https://github.com/jessrosenfield. Do you know if it fails on the same line of code every time? That line is simply setting a variable to None and is only ever called by a single thread, so I can't see how it would cause a SegmentationFault.

My suspicion is that this is instead an out-of-memory error. Since you said you are running this in a Docker container, can you check that the Docker engine is configured to allocate a sufficient amount of memory to the container? Try 32 GB - that should be overkill, but once you figure out whether that is the fix you can start scaling back, perhaps using a profiler to determine the peak usage.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jdidion/atropos/issues/88?email_source=notifications&email_token=AALXJPNQIY2XCGXGD5NO343Q2O2S7A5CNFSM4J6ZA4M2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHUSBAA#issuecomment-568926336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALXJPKGHYOGFL7VV44RVZ3Q2O2S7ANCNFSM4J6ZA4MQ .

jessrosenfield commented 4 years ago

I'll also check that the trace is consistent.

On Mon, Dec 30, 2019, 12:38 Jessica Rosenfield rosenfield.jess@gmail.com wrote:

Sorry for the delay, I've been on vacation. This was already run with excessive memory (64GB) allocated on the ec2 instance and the docker process was only using a small fraction of that memory. When I return I will do some profiling to check that the docker container has proper access to the memory on the instance and do additional profiling of memory used over time.

On Wed, Dec 25, 2019, 13:51 John Didion notifications@github.com wrote:

Thanks @jessrosenfield https://github.com/jessrosenfield. Do you know if it fails on the same line of code every time? That line is simply setting a variable to None and is only ever called by a single thread, so I can't see how it would cause a SegmentationFault.

My suspicion is that this is instead an out-of-memory error. Since you said you are running this in a Docker container, can you check that the Docker engine is configured to allocate a sufficient amount of memory to the container? Try 32 GB - that should be overkill, but once you figure out whether that is the fix you can start scaling back, perhaps using a profiler to determine the peak usage.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jdidion/atropos/issues/88?email_source=notifications&email_token=AALXJPNQIY2XCGXGD5NO343Q2O2S7A5CNFSM4J6ZA4M2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHUSBAA#issuecomment-568926336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALXJPKGHYOGFL7VV44RVZ3Q2O2S7ANCNFSM4J6ZA4MQ .

siddharthab commented 4 years ago

I was able to inspect the stack trace in the core dump and was able to trace it to a bug in python 3.5.2 (default on ubuntu 16.04). This bug has been fixed from python 3.5.3 onwards.

This issue can be closed.

jdidion commented 4 years ago

Thank you @siddharthab! I will make a note of this on the readme so that others avoid 3.5.2.

Starting from atropos 2.0, python 3.6 will be required at a minimum, so this won't be an issue anymore.