Bus error/seg fault when using flat on sliced, memmap'd array (Trac #2148)

numpy-gitbot commented 11 years ago

Original ticket http://projects.scipy.org/numpy/ticket/2148 on 2012-05-31 by trac user farrowch, assigned to unknown.

The following code crashes on numpy 1.6.1, on OSX 10.6.

import numpy as np
shape = (8, 8)
dtype = np.dtype(np.uint8)
image = np.random.randint(0, 256, shape).astype(dtype)
image.tofile("test_image.bin")
image = np.memmap("test_image.bin", dtype=dtype, shape=shape, mode='r')
arr = image[::2,::2]
np.sum(arr.flat)

Here's what I've discovered so far about this:

the bus error only occurs with mode 'r'
the dimensionality of the array appears to be irrelevant
if the array slice does not change the strides, the bus error does not occur
no 'arr.flat', no bus error
Other aggregating functions (e.g. fmin.reduce) will induce the error
Iterating over arr.flat will not cause a bus error

Based on this, I suspect the issue is with the C-facing side of the flat iterator.

numpy-gitbot commented 11 years ago

trac user farrowch wrote on 2012-05-31

Contributed by Val Kalatsky:

Confirmed on Ubuntu, np.version 1.5.1 and 1.6.1 (backtraces are bellow).
Something seems to be broken before it comes to memcpy and/or _aligned_contig_to_strided_size1.

np.version 1.6.1

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff5b63cdf in _aligned_contig_to_strided_size1 ()
   from /home/vkalatsky/src/epd8/lib/python2.7/site-packages/numpy/core/multiarray.so
(gdb) bt
#0  0x00007ffff5b63cdf in _aligned_contig_to_strided_size1 ()
   from /home/vkalatsky/src/epd8/lib/python2.7/site-packages/numpy/core/multiarray.so
#1  0x00007ffff5bc2c31 in PyArray_CopyAnyIntoOrdered ()
   from /home/vkalatsky/src/epd8/lib/python2.7/site-packages/numpy/core/multiarray.so
#2  0x00007ffff5bc3329 in array_dealloc () from /home/vkalatsky/src/epd8/lib/python2.7/site-packages/numpy/core/multiarray.so
#3  0x00007ffff71b7bbb in meth_dealloc (m=0xb6fdd0) at Objects/methodobject.c:134
#4  0x00007ffff720df99 in PyEval_EvalFrameEx (f=0xc3f310, throwflag=<value optimized out>) at Python/ceval.c:2712
#5  0x00007ffff7214722 in PyEval_EvalCodeEx (co=0x791d30, globals=<value optimized out>, locals=<value optimized out>, args=0xab7fd0, 
    argcount=5, kws=0xab7ff8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#6  0x00007ffff72126b6 in call_function (f=0xab7e20, throwflag=<value optimized out>) at Python/ceval.c:4109
#7  PyEval_EvalFrameEx (f=0xab7e20, throwflag=<value optimized out>) at Python/ceval.c:2666
#8  0x00007ffff7214722 in PyEval_EvalCodeEx (co=0x7988b0, globals=<value optimized out>, locals=<value optimized out>, args=0x3, 
    argcount=1, kws=0x3, kwcount=0, defs=0x79c518, defcount=3, closure=0x0) at Python/ceval.c:3253
#9  0x00007ffff72126b6 in call_function (f=0x6d0820, throwflag=<value optimized out>) at Python/ceval.c:4109
#10 PyEval_EvalFrameEx (f=0x6d0820, throwflag=<value optimized out>) at Python/ceval.c:2666
#11 0x00007ffff7214722 in PyEval_EvalCodeEx (co=0x7ffff7eb41b0, globals=<value optimized out>, locals=<value optimized out>, args=0x0, 
    argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#12 0x00007ffff7214772 in PyEval_EvalCode (co=0x7ffff7ff5000, globals=0x2, locals=0x897cb1) at Python/ceval.c:667
#13 0x00007ffff722e432 in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=0x640140, locals=0x640140, 
    flags=<value optimized out>, arena=<value optimized out>) at Python/pythonrun.c:1346
#14 0x00007ffff722e506 in PyRun_FileExFlags (fp=0x6cb390, filename=0x7fffffffe599 "cf_test.py", start=257, globals=0x640140, 
    locals=0x640140, closeit=1, flags=0x7fffffffe1a0) at Python/pythonrun.c:1332
#15 0x00007ffff722fa67 in PyRun_SimpleFileExFlags (fp=<value optimized out>, filename=0x7fffffffe599 "cf_test.py", closeit=1, 
    flags=0x7fffffffe1a0) at Python/pythonrun.c:936
#16 0x00007ffff72401e2 in Py_Main (argc=1, argv=0x7fffffffe2c8) at Modules/main.c:689
#17 0x00007ffff652cd8e in __libc_start_main (main=<value optimized out>, argc=<value optimized out>, ubp_av=<value optimized out>, 
    init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fffffffe2b8)
    at libc-start.c:226
#18 0x00000000004006f9 in _start ()

np.version 1.5.1

Program received signal SIGSEGV, Segmentation fault.
memcpy () at ../sysdeps/x86_64/memcpy.S:67
67  ../sysdeps/x86_64/memcpy.S: No such file or directory.
    in ../sysdeps/x86_64/memcpy.S
(gdb) bt
#0  memcpy () at ../sysdeps/x86_64/memcpy.S:67
#1  0x00007ffff5bf2460 in PyArray_CopyAnyInto () from /home/vkalatsky/epd70/lib/python2.7/site-packages/numpy/core/multiarray.so
#2  0x00007ffff5bf2a69 in array_dealloc () from /home/vkalatsky/epd70/lib/python2.7/site-packages/numpy/core/multiarray.so
#3  0x00007ffff71b889b in meth_dealloc (m=0xbdd170) at Objects/methodobject.c:134
#4  0x00007ffff720e8b9 in PyEval_EvalFrameEx (f=0xc12450, throwflag=<value optimized out>) at Python/ceval.c:2711
#5  0x00007ffff7215042 in PyEval_EvalCodeEx (co=0x81e630, globals=<value optimized out>, locals=<value optimized out>, args=0x7bd820, 
    argcount=5, kws=0x7bd848, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#6  0x00007ffff7212fd6 in call_function (f=0x7bd670, throwflag=<value optimized out>) at Python/ceval.c:4108
#7  PyEval_EvalFrameEx (f=0x7bd670, throwflag=<value optimized out>) at Python/ceval.c:2665
#8  0x00007ffff7215042 in PyEval_EvalCodeEx (co=0x85f1b0, globals=<value optimized out>, locals=<value optimized out>, args=0x3, 
    argcount=1, kws=0x3, kwcount=0, defs=0x85e928, defcount=3, closure=0x0) at Python/ceval.c:3252
#9  0x00007ffff7212fd6 in call_function (f=0x747ea0, throwflag=<value optimized out>) at Python/ceval.c:4108
#10 PyEval_EvalFrameEx (f=0x747ea0, throwflag=<value optimized out>) at Python/ceval.c:2665
#11 0x00007ffff7215042 in PyEval_EvalCodeEx (co=0x691b30, globals=<value optimized out>, locals=<value optimized out>, args=0x0, 
    argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#12 0x00007ffff7215092 in PyEval_EvalCode (co=0x7ffff7ff3000, globals=0xb55550, locals=0x1) at Python/ceval.c:666
#13 0x00007ffff722eb82 in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=0x640140, locals=0x640140, 
    flags=<value optimized out>, arena=<value optimized out>) at Python/pythonrun.c:1346
#14 0x00007ffff722ec56 in PyRun_FileExFlags (fp=0x743250, filename=0x7fffffffe5bd "cf_test.py", start=257, globals=0x640140, 
    locals=0x640140, closeit=1, flags=0x7fffffffe1c0) at Python/pythonrun.c:1332
#15 0x00007ffff72301b7 in PyRun_SimpleFileExFlags (fp=<value optimized out>, filename=0x7fffffffe5bd "cf_test.py", closeit=1, 
    flags=0x7fffffffe1c0) at Python/pythonrun.c:936
#16 0x00007ffff724083a in Py_Main (argc=1, argv=0x7fffffffe2e8) at Modules/main.c:676
#17 0x00007ffff652dd8e in __libc_start_main (main=<value optimized out>, argc=<value optimized out>, ubp_av=<value optimized out>, 
    init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fffffffe2d8)
    at libc-start.c:226
#18 0x00000000004006f9 in _start ()

numpy-gitbot commented 11 years ago

atmention:mwiebe wrote on 2012-05-31

It looks like the problem is that the flat iterator generates an UPDATEIFCOPY array whenever an array version of it is requested, and that leads to the following sequence of events:

Operation requests array from flatiter
Flatiter gives back UPDATEIFCOPY array
Operation completes, triggers UPDATEIFCOPY
Segfault because memmap marked the memory as read only at the OS level

This simpler version also triggers it for me:

import numpy as np
shape = (8, 8)
dtype = np.dtype(np.uint8)
image = np.random.randint(0, 256, shape).astype(dtype)
image.tofile("test_image.bin")
image = np.memmap("test_image.bin", dtype=dtype, shape=shape, mode='r')
arr = image[::2,::2]
temp = np.asarray(arr)
del(temp) # triggers UPDATEIFCOPY, crash

numpy-gitbot commented 11 years ago

atmention:mwiebe wrote on 2012-05-31

Oops - that should be np.asarray(arr.flat), sorry

numpy-gitbot commented 11 years ago

atmention:WarrenWeckesser wrote on 2012-06-01

I'm getting burned by this in a project that I'm updating from numpy 1.5.1 to 1.6.1. The simplified code that triggers the crash is:

x = np.memmap('stuff.dat', dtype=np.uint8, mode='r')
y = x[::2]
print np.nanmin(y)   # Segfaults

This does not segfault in 1.5.1.

numpy-gitbot commented 11 years ago

atmention:teoliphant wrote on 2012-06-08

This segfault should be fixed by PR 280 and 299 which fix the problems of both setting to a readonly base with UPDATEIFCOPY and with flattened iterator objects able to be used to create arrays which write to read-only memory.

thouis / numpy-trac-migration

Bus error/seg fault when using flat on sliced, memmap'd array (Trac #2148) #5940