stride_tricks with float16 (Trac #2073)

[numpy-gitbot]

Original ticket http://projects.scipy.org/numpy/ticket/2073 on 2012-03-05 by trac user npinto, assigned to unknown.

It looks like stride_tricks don't work with float16:

% python bug.py                                                                                                                                                                                                                                                                                                    (develop…)
trying with float32
trying with float16
Traceback (most recent call last):
  File "bug.py", line 18, in <module>
    arr_out = as_strided(arr_in, shape=new_shape, strides=new_strides)
  File "/usr/lib64/python2.6/site-packages/numpy/lib/stride_tricks.py", line 28, in as_strided
    return np.asarray(DummyArray(interface, base=x))
  File "/usr/lib64/python2.6/site-packages/numpy/core/numeric.py", line 235, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: unsupported typestring

[numpy-gitbot]

Attachment added by trac user npinto on 2012-03-05: bug.py

[numpy-gitbot]

atmention:mwiebe wrote on 2012-03-12

I suspect the stride tricks code has hard-coded knowledge about the numpy dtypes. This should be changed so it directly uses the information provided by the dtype object instead, so that it would work with third-party extension dtypes, too.

This should be a relatively straightforward Python-level code fix, anyone with a decent knowledge of Python should be able to tackle this bug. Too bad there's no way to mark that in a nice searchable field yet.

[numpy-gitbot]

trac user npinto wrote on 2012-03-12

I was able to nail and reproduce the bug in bug2.py:

% cat bug2.py                                                                                         (master⚡)
import numpy as np

f2 = np.empty((1), dtype='float16')

class Foo: pass
a = Foo()
a.__array_interface__ = f2.__array_interface__

np.array(a, 'float16')
np.array(f2, 'float16')

% python ipython_log.py
Traceback (most recent call last):
  File "ipython_log.py", line 9, in <module>
    np.array(a, 'float16')
ValueError: unsupported typestring

[numpy-gitbot]

Attachment added by trac user npinto on 2012-03-12: bug2.py

[numpy-gitbot]

trac user npinto wrote on 2012-03-12

What does need to be done to properly "inherit" the array interface ?

[numpy-gitbot]

atmention:mwiebe wrote on 2012-03-12

Which version of NumPy are you using? e.g.

In [19]: np.__version__
Out[19]: '1.7.0.dev-Unknown'

[numpy-gitbot]

trac user npinto wrote on 2012-03-12

python -c 'import numpy as np; print np.__version__'
1.6.0

ps: trac's interface + captcha are awful... any hope to get the bug tracker on github?

[numpy-gitbot]

atmention:mwiebe wrote on 2012-03-12

Re: "inherit" the array interface: I'm not sure. You should probably ask this question on the mailing list.

Definitely agree about trac. Travis has gotten NumPy approved for jetbrains youtrack to try it out, this test will happen as he or someone else has the time to get that running. The github has many problems of its own, especially for how to prioritize and manipulate mass bugs at once, so I'd rather not switch to github either.

When I run this, it works for me. I guess there's something up with the dtype construction in NumPy 1.6 that has been fixed in 1.7:

In [18]: np.array([1.5], dtype='float16')
Out[18]: array([ 1.5], dtype=float16)

[numpy-gitbot]

atmention:mwiebe wrote on 2012-03-12

Sorry, I didn't look closely enough at your reproduction of the bug. Thanks for doing that, this looks like a problem in the array_interface code (incidentally an interface I'm not a big fan of, but of course it still has its uses for the time being).

What I originally said about stride_tricks not using the dtype metadata information properly seems to apply to the array_interface code.

[numpy-gitbot]

trac user npinto wrote on 2012-03-12

Reproducing the bug with HEAD:

In [1]: import numpy as np

In [2]: np.__version__
Out[2]: '1.7.0.dev-737f690'

In [3]: !cat bug2.py
import numpy as np

f2 = np.empty((1), dtype='float16')

class Foo: pass
a = Foo()
a.__array_interface__ = f2.__array_interface__

np.array(a, 'float16')
np.array(f2, 'float16')

In [4]: run bug2.py
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/Users/nich2o/venv/numpy-dev/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    173             else:
    174                 filename = fname
--> 175             __builtin__.execfile(filename, *where)

/Users/nich2o/bug2.py in <module>()
      7 a.__array_interface__ = f2.__array_interface__
      8 
----> 9 np.array(a, 'float16')
     10 np.array(f2, 'float16')

ValueError: unsupported typestring

[numpy-gitbot]

atmention:mwiebe wrote on 2012-03-12

Here's the guilty code if you want to try hacking on it:

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/common.c#L482

This code should not be doing this low-level stuff, it should rather be calling an appropriate dtype descriptor constructor to handle the proper conversion.

[numpy-gitbot]

trac user npinto wrote on 2012-03-12

I guess I could add NPY_HALF here: https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/common.c#L558

But it won't solve the problem. What is the appropriate "dtype descriptor constructor" to call ?

[numpy-gitbot]

atmention:mwiebe wrote on 2012-03-14

It should be fine to call the public-facing one which is documented here:

http://docs.scipy.org/doc/numpy/reference/c-api.array.html#PyArray_DescrConverter

This is probably pretty close to what it should look like (I haven't tried to compile or run this):

    PyArray_Descr *dtype = NULL;
    PyObject *stringobj = PyString_FromString(c_str)
    if (stringobj == NULL) {
        return NULL;
    }
    if (PyArray_DescrConverter(stringobj, &dtype) != NPY_SUCCEED)
        Py_DECREF(stringobj);
        return NULL;
    }
    Py_DECREF(stringobj);
    return dtype;

[numpy-gitbot]

trac user npinto wrote on 2012-03-14

Thanks a ton. I added your code and a quick test a PR on github: https://github.com/numpy/numpy/pull/234

HTH

[numpy-gitbot]

atmention:charris wrote on 2012-03-31

Should be fixed in dbf2351..d959ab9.

[numpy-gitbot]

Milestone changed to NumPy 1.7 by atmention:charris on 2012-03-31