search

robertwb / issues-import-test

0 stars 0 forks source link

memoryviews of arrays of structs fail to work in some cases #86

Open robertwb opened 8 years ago

robertwb commented 8 years ago

Reported by scoder on 29 Jun 2015 13:40 UTC As reported on the mailing list:

There seem to be some bugs regarding memoryviews of structured arrays in Cython when the struct itself contains arrays. I have a Cython fix for one of them (the most relevant to me), but first I would like to make sure that these are really bugs and I am not doing anything stupid. This question is related to http://stackoverflow.com/questions/17239091/cython-memoryviews-from-array-of-structs, but the answer there works only for char arrays. An array of doubles breaks it again.

It seems to me that Cython implementation solves some special cases, but not the general one. My proposed fix would be to insert

ndim = ctx->head->field->type->ndim;

after

/* Process the previous element */

if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;

in Cython\Utility\Buffer.c

This fixes tests 2, 4 and 7, but not 8 and 9 (see below). I am not sure about the details of what the code in Buffer.c is supposed to do and whether my modification would break some other functionality, but my code (not only the tests) works with this fix.

Or am I possibly simply missing "the right way" to handle arrays of structs containing arrays in Cython? But in either case the compiler shouldn't crash like in test 9...

Here is the code of my tests with and without numpy with results ranging from "ok" to "compiler crash". The numpy tests require these imports:

import numpy as np
cimport numpy as np

*Test 1 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*Test 2 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value*[test():
    cdef My_Struct my_struct
    cdef My_Struct[:](1]*
cpdef) my_view = <My_Struct[Expected 1 dimension(s), got 1*
*
*
*Test 3 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*1*](:1]>&my_struct

*ValueError:)
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 4 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*2*](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*
*
*
*Test 5 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    *char* my_second_value[2](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','*a*',2)](('my_first_value','float64'),), 
align=False)
    cdef My_Struct[my_view = my_array

*Test 6 (OK)*
cdef packed struct My_Array_Struct:
    double values[*1*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 7 (FAILS)*
cdef packed struct My_Array_Struct:
    double values[*2*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*

*Test 8 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value*[1](:])*
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*ValueError: Buffer dtype mismatch, expected 'double' but got end in 
'My_Struct.my_second_value'*

*Test 9 (COMPILER CRASH)*
cdef packed struct My_Array_Struct:
    double values[packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value*[1](2]
cdef)*
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

Migrated-From: http://trac.cython.org/ticket/853

robertwb commented 8 years ago

Modified by scoder on 29 Jun 2015 13:43 UTC