memoryviews of arrays of structs fail to work in some cases

As reported on the mailing list:

There seem to be some bugs regarding memoryviews of structured arrays in Cython when the struct itself contains arrays. I have a Cython fix for one of them (the most relevant to me), but first I would like to make sure that these are really bugs and I am not doing anything stupid. This question is related to http://stackoverflow.com/questions/17239091/cython-memoryviews-from-array-of-structs, but the answer there works only for char arrays. An array of doubles breaks it again.

It seems to me that Cython implementation solves some special cases, but not the general one. My proposed fix would be to insert

ndim = ctx->head->field->type->ndim;

after

/* Process the previous element */

if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;

in Cython\Utility\Buffer.c

This fixes tests 2, 4 and 7, but not 8 and 9 (see below). I am not sure about the details of what the code in Buffer.c is supposed to do and whether my modification would break some other functionality, but my code (not only the tests) works with this fix.

Or am I possibly simply missing "the right way" to handle arrays of structs containing arrays in Cython? But in either case the compiler shouldn't crash like in test 9...

Here is the code of my tests with and without numpy with results ranging from "ok" to "compiler crash". The numpy tests require these imports:

import numpy as np
cimport numpy as np

*Test 1 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*Test 2 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value*[test():
    cdef My_Struct my_struct
    cdef My_Struct[:](1]*
cpdef) my_view = <My_Struct[Expected 1 dimension(s), got 1*
*
*
*Test 3 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*1*](:1]>&my_struct

*ValueError:)
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 4 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*2*](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*
*
*
*Test 5 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    *char* my_second_value[2](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','*a*',2)](('my_first_value','float64'),), 
align=False)
    cdef My_Struct[my_view = my_array

*Test 6 (OK)*
cdef packed struct My_Array_Struct:
    double values[*1*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 7 (FAILS)*
cdef packed struct My_Array_Struct:
    double values[*2*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*

*Test 8 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value*[1](:])*
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*ValueError: Buffer dtype mismatch, expected 'double' but got end in 
'My_Struct.my_second_value'*

*Test 9 (COMPILER CRASH)*
cdef packed struct My_Array_Struct:
    double values[packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value*[1](2]
cdef)*
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

Migrated from http://trac.cython.org/ticket/853

scoder changed description from

As reported on the mailing list:

It seems to me that Cython implementation solves some special cases, but not the general one. My proposed fix would be to insert

ndim = ctx->head->field->type->ndim;

after

/* Process the previous element */

if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;

in Cython\Utility\Buffer.c

Or am I possibly simply missing "the right way" to handle arrays of structs containing arrays in Cython? But in either case the compiler shouldn't crash like in test 9...

Here is the code of my tests with and without numpy with results ranging from "ok" to "compiler crash". The numpy tests require these imports:

import numpy as np
cimport numpy as np

*Test 1 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*Test 2 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value*[test():
    cdef My_Struct my_struct
    cdef My_Struct[:](1]*
cpdef) my_view = <My_Struct[Expected 1 dimension(s), got 1*
*
*
*Test 3 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*1*](:1]>&my_struct

*ValueError:)
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 4 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*2*](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*
*
*
*Test 5 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    *char* my_second_value[2](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','*a*',2)](('my_first_value','float64'),), 
align=False)
    cdef My_Struct[my_view = my_array

*Test 6 (OK)*
cdef packed struct My_Array_Struct:
    double values[*1*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 7 (FAILS)*
cdef packed struct My_Array_Struct:
    double values[*2*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*

*Test 8 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value*[1](:])*
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*ValueError: Buffer dtype mismatch, expected 'double' but got end in 
'My_Struct.my_second_value'*

*Test 9 (COMPILER CRASH)*
cdef packed struct My_Array_Struct:
    double values[packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value*[1](2]
cdef)*
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct
``` to

As reported on the mailing list:

There seem to be some bugs regarding memoryviews of structured arrays 
in Cython when the struct itself contains arrays. I have a Cython fix for 
one of them (the most relevant to me), but first I would like to make sure 
that these are really bugs and I am not doing anything stupid. This 
question is related to 
http://stackoverflow.com/questions/17239091/cython-memoryviews-from-array-of-structs, 
but the answer there works only for char arrays. An array of doubles breaks 
it again.

It seems to me that Cython implementation solves some special cases, but 
not the general one. My proposed fix would be to insert

ndim = ctx->head->field->type->ndim;

after

/* Process the previous element */

if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;

in Cython\Utility\Buffer.c

This fixes tests 2, 4 and 7, but not 8 and 9 (see below). I am not sure 
about the details of what the code in Buffer.c is supposed to do and 
whether my modification would break some other functionality, but my code 
(not only the tests) works with this fix.

Or am I possibly simply missing "the right way" to handle arrays of structs 
containing arrays in Cython? But in either case the compiler shouldn't 
crash like in test 9...

Here is the code of my tests with and without numpy with results ranging 
from "ok" to "compiler crash".
The numpy tests require these imports:

import numpy as np cimport numpy as np

Test 1 (OK) cdef packed struct My_Struct: double my_first_value double my_second_value cpdef test(): cdef My_Struct my_struct cdef My_Struct[my_view = <My_Struct:1>&my_struct

Test 2 (FAILS) cdef packed struct My_Struct: double my_first_value double my_secondvalue[test(): cdef My_Struct my_struct cdef My_Struct: my_view = <MyStruct[Expected 1 dimension(s), got 1 Test 3 (OK) cdef packed struct My_Struct: double my_first_value double my_second_value[1](:1]>&my_struct

ValueError:) cpdef test(): cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype([('my_second_value','float64', 1*)](%28'my_first_value','float64'%29,), align=False) cdef My_Struct[my_view = my_array

Test 4 (FAILS) cdef packed struct My_Struct: double my_first_value double my_second_value2 cpdef test(): cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype(('my_second_value','float64', 2), align=False) cdef My_Struct[my_view = my_array

ValueError: Expected 1 dimension(s), got 1 Test 5 (OK) cdef packed struct My_Struct: double my_first_value char my_second_value2 cpdef test(): cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype(('my_second_value','a',2), align=False) cdef My_Struct[my_view = my_array

Test 6 (OK) cdef packed struct My_Array_Struct: double values1 cdef packed struct My_Struct: double my_first_value My_Array_Struct my_second_value cpdef test(): cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype(('my_second_value','float64', 1), align=False) cdef My_Struct[my_view = my_array

Test 7 (FAILS) cdef packed struct My_Array_Struct: double values2 cdef packed struct My_Struct: double my_first_value My_Array_Struct my_second_value cpdef test(): cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype(('my_second_value','float64', 2), align=False) cdef My_Struct[my_view = my_array

ValueError: Expected 1 dimension(s), got 1

Test 8 (FAILS) cdef packed struct My_Struct: double my_firstvalue1_ double my_second_value cpdef test(): cdef My_Struct my_struct cdef My_Struct[my_view = <My_Struct:1>&my_struct

_ValueError: Buffer dtype mismatch, expected 'double' but got end in 'My_Struct.my_secondvalue'

Test 9 (COMPILER CRASH) cdef packed struct My_Array_Struct: double values[packed struct My_Struct: double my_first_value My_Array_Struct my_secondvalue1_ cpdef test(): cdef My_Struct my_struct cdef My_Struct[my_view = <My_Struct:1>&my_struct

robertwb / issues-import-test

memoryviews of arrays of structs fail to work in some cases #1005