Segfault on umath.cpython-32mu.so (Trac #2129)

[numpy-gitbot]

Original ticket http://projects.scipy.org/numpy/ticket/2129 on 2012-05-08 by trac user Sachiel, assigned to unknown.

On a code involving Numpy, Matplotlib and Netcdf4, I'm getting a Segmentation Fault from numpy/core/umath.cpython-32mu.so during the execution of the pyplot.show() funcion. AT first I thought it was a problem from Matplotlib, but given that the segfaulted library it's from Numpy, I decided to report it here.

System: Python 3.2.3 Numpy 1.6.1 Matplotlib 1.1.0 Python-NetCDF4 0.9.9

[numpy-gitbot]

Attachment added by trac user Sachiel on 2012-05-08: backtrace_python.txt

[numpy-gitbot]

atmention:charris wrote on 2012-05-08

The backtrace doesn't help much. It looks like you are using a distro provided numpy, could you try with current development? It should provide a better backtrace if the segfault persists.

It would also help if you could put together a script that exhibits the problem. I know that is tricky with so many parts involved...

[numpy-gitbot]

Attachment added by trac user Sachiel on 2012-05-09: backtrace_python_new.txt

[numpy-gitbot]

trac user Sachiel wrote on 2012-05-09

I updated Numpy to the Git version:

numpy.version '1.7.0.dev-Unknown'

And the problem persists. I attach the new backtrace. I'll try to create a "short" code to reproduce the problem, independent from the data. Meanwhile, I try to help describing what it does:

• Load 45 arrays, of 4600x3840, from netcdf files.
• Plot 15 RGB images, composed by 3 arrays each, on the same figure.

I think it is a memory related issue, as it eats all my RAM and swap, but I was expecting a Memory Exception instead of a Segfault. If I plot grayscale images instead of RGB, so loading just 15 arrays (one per image), it works OK.

BTW, my system is: Arch Linux Linux mirto 3.3.4-2-ARCH #1 SMP PREEMPT Wed May 2 18:28:42 CEST 2012 x86_64 Intel(R) Core(TM) i5-2500 CPU atmention: 3.30GHz GenuineIntel GNU/Linux 4 GB RAM, 4GB swap

[numpy-gitbot]

Milestone changed to NumPy 1.7 by trac user Sachiel on 2012-05-09

[numpy-gitbot]

atmention:charris wrote on 2012-05-09

Thanks for the update. It looks like you need to compile with the -g option in CFLAGS so the backtrace will show a bit more info. That probably means cleaning out the build directory first as distutils doesn't track option changes.

You shouldn't mark the ticket for review unless you have a patch fixing the problem.

[numpy-gitbot]

Attachment added by trac user Sachiel on 2012-05-10: backtrace_python_debug.txt

[numpy-gitbot]

trac user Sachiel wrote on 2012-05-10

Uploaded new backtrace, but I think not much extra info is provided (maybe I compiled it incorrectly?). Any way to check the compilation flags once it is done? I did it from zero, didn't reuse the already downloaded source.

[numpy-gitbot]

atmention:charris wrote on 2012-05-10

I hoping to fill in the missing '??'.

#0  0x00007ffff5bb5934 in ?? () from /usr/lib/python3.2/site-packages/numpy/core/umath.cpython-32mu.so

This should work, but maybe you did this, although you say CFLAG above

export CFLAGS='-g'

The flag needs to be there during both compilation of numpy and linking.

I notice that you seem to be running on 64 bit linux but the umath library looks like 32 bits. Is that correct?

My own guess is that the problem is in Python-NetCDF4 because that is the only component that isn't widely tested. But it would still be nice to see where the problem is. It looks like getting a debug package installed on archlinux is a bit of a job: https://wiki.archlinux.org/index.php/Debug_-_Getting_Traces

[numpy-gitbot]

trac user Sachiel wrote on 2012-05-11

I redownloaded the source and build it following the Arch Wiki guide for debugging, and now I am getting a Memory Error from Matplotlib, which puzzles me. That's the kind of error I was expecting (plotting such huge amount of data), but I don't get why compiling numpy with a different optimization level and debug flags would lead to segfault to memory error.

I'll play a bit more with it, installing again with no debug flags, etc., to be sure that I can reproduce the scenarios and it is not affected by other system components.

[numpy-gitbot]

trac user Sachiel wrote on 2012-05-11

When running it from more lightweight DE (LXDE instead of Gnome), I get the segfault again instead of the Memory Exception. I attach the new backtrace.

[numpy-gitbot]

Attachment added by trac user Sachiel on 2012-05-11: backtrace_python_debug_notstrip.txt

[numpy-gitbot]

atmention:charris wrote on 2012-05-12

Hmm, that's beginning to look interesting. Does the segfault happen immediately, or does it take a while? I'm trying to get a handle on whether the loop usually works, or just fails immediately.

[numpy-gitbot]

atmention:charris wrote on 2012-05-12

I think you are correct that a memory error is the source of the problem. The first part of the back trace follows a well used path, so what I think is happening is that there was a memory error that wasn't caught and a NULL pointer got passed down the call stack.

[numpy-gitbot]

trac user Sachiel wrote on 2012-05-15

Just loading and plotting data for 6 RGB images instead of 15, doesn't provoke the segfault.

I've also discovered that if I delete each array after plotting it but before rendering on the screen (so, after plt.imshow() but before plt.show()), it works OK. So, definitely, it's about and excessive usage of memory.

About your question if it happens immediately, well, the program takes it's time to read the data from the HD, and then it becomes slow as it start using swap. At one point an empty (all grey) figure window appears, and then the Segmentation Fault message on the console.

[numpy-gitbot]

Milestone changed to Unscheduled by atmention:rgommers on 2012-05-20