search

liberfa / pyerfa

Python bindings for ERFA routines
https://pyerfa.readthedocs.io
BSD 3-Clause "New" or "Revised" License
34 stars 24 forks source link

ValueError: Invalid data-type for array with Python 3.11 #97

Closed saimn closed 1 year ago

saimn commented 1 year ago

While installing pyerfa from source (as an Astropy dependency) on Python 3.11, I get this error with the ufunc calls. For exemple, running tests from the installed package results in these errors: 

❯ pytest ~/.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests

=================================== short test summary info ====================================
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_erfa_wrapper - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_errwarn_reporting - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_vector_inouts - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_rz - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_float32_input - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_set_leap_seconds - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_validation[table0-January] - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_validation[table1-jump] - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_validation[table2-dimension] - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_update_leap_seconds - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ab - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2ibpn - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2ixy - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2tcio - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2teqx - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_cal2jd - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_dat - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_eform - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_eors - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_epj - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gc2gd - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gc2gde - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gd2gc - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gst06 - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ld - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ldsun - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_num00b - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_num06a - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_pmpx - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ppsp - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_pr00 - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_rx - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ry - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_rz - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_sxp - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_tporv - ValueError: Invalid data-type for array
FAILED ../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_tpstv - ValueError: Invalid data-type for array
==================== 37 failed, 236 passed, 5 xfailed, 22 warnings in 3.10s ====================

Failure example:

__________________________________________ test_tporv __________________________________________

    def test_tporv():
        v = np.empty((3,), float)
        vz1 = np.empty((3,), float)
        vz2 = np.empty((3,), float)
        xi = -0.03
        eta = 0.07
        ra = 1.3
        dec = 1.5
        v = erfa_ufunc.s2c(ra, dec)
>       vz1, vz2, n = erfa_ufunc.tporv(xi, eta, v)
E       ValueError: Invalid data-type for array

../../../.pyenv/versions/3.11.0/lib/python3.11/site-packages/erfa/tests/test_ufunc.py:3511: ValueError

Astropy's CI on Python 3.11 is not failing, so I don't know, maybe something due to the GCC version ?

❯ gcc --version                                           
gcc (GCC) 12.2.0

When building from source I get many "incompatible pointer type" warnings, and a "dangling pointer" one:

      |         void (*)(char **, npy_intp *, npy_intp *, void *) {aka void (*)(char **, long int *, long int *, void *)}
  erfa/ufunc.c:14046:9: note: expected ‘PyUFuncGenericFunction’ {aka ‘void (*)(char **, const long int *, const long int *, void *)’} but argument is of type ‘void (*)(char **, npy_intp *, npy_intp *, void *)’ {aka ‘void (*)(char **, long int *, long int *, void *)’}
  erfa/ufunc.c:14080:9: warning: passing argument 3 of ‘(int (*)(PyUFuncObject *, PyArray_Descr *, void (*)(char **, const npy_intp *, const npy_intp *, void *), PyArray_Descr **, void *))*(PyUFunc_API + 328)’ from incompatible pointer type [-Wincompatible-pointer-types]
  14080 |         ufunc_loop_sxpv, dtypes, NULL);
        |         ^~~~~~~~~~~~~~~
        |         |
        |         void (*)(char **, npy_intp *, npy_intp *, void *) {aka void (*)(char **, long int *, long int *, void *)}
  erfa/ufunc.c:14080:9: note: expected ‘PyUFuncGenericFunction’ {aka ‘void (*)(char **, const long int *, const long int *, void *)’} but argument is of type ‘void (*)(char **, npy_intp *, npy_intp *, void *)’ {aka ‘void (*)(char **, long int *, long int *, void *)’}
  erfa/ufunc.c: In function ‘ErfaUFuncTypeResolver’:
  erfa/ufunc.c:9729:16: warning: dangling pointer ‘types’ to ‘types_array’ may be used [-Wdangling-pointer=]
   9729 |         return set_ufunc_loop_data_types(ufunc, operands, out_dtypes,
        |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   9730 |                                          types, dtypes);
        |                                          ~~~~~~~~~~~~~~
  erfa/ufunc.c:9715:13: note: ‘types_array’ declared here
   9715 |         int types_array[NPY_MAXARGS];
        |             ^~~~~~~~~~~
jobovy commented 1 year ago

Just ran these tests in a clean Python 3.11 conda environment with the latest pyerfa release on an M1 Mac and the tests all run fine, so perhaps this could be closed?

EDIT: when compiling from source, the tests also pass for me.

avalentino commented 1 year ago

Thanks @jobovy, closing. @saimn please feel free to re-open if things still do not work for you.

saimn commented 1 year ago

Still fails if I install from source:

=================================================================================== short test summary info ====================================================================================
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_erfa_wrapper - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_errwarn_reporting - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_vector_inouts - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_rz - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::test_float32_input - numpy.core._exceptions._UFuncNoLoopError: ufunc 'p2s' did not contain a loop with signature matching types <class 'numpy.dtype[float32]'> -> (None, None, None)
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_set_leap_seconds - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_validation[table0-January] - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_validation[table1-jump] - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_validation[table2-dimension] - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_erfa.py::TestLeapSeconds::test_update_leap_seconds - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ab - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2ibpn - numpy.core._exceptions._UFuncNoLoopError: ufunc 'c2ibpn' did not contain a loop with signature matching types (<class 'numpy.dtype[bool_]'>, <class 'numpy.dtype[bool_]'>, <class 'numpy.dt...
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2ixy - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2tcio - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_c2teqx - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_cal2jd - numpy.core._exceptions._UFuncNoLoopError: ufunc 'cal2jd' did not contain a loop with signature matching types (<class 'numpy.dtype[int64]'>, <class 'numpy.dtype[int64]'>, <class 'numpy.dt...
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_dat - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_eform - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_eors - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_epj - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gc2gd - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gc2gde - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gd2gc - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_gst06 - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ld - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ldsun - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_num00b - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_num06a - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_pmpx - numpy.core._exceptions._UFuncNoLoopError: ufunc 'pmpx' did not contain a loop with signature matching types (<class 'numpy.dtype[float64]'>, <class 'numpy.dtype[float64]'>, <class 'numpy....
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ppsp - numpy.core._exceptions._UFuncNoLoopError: ufunc 'ppsp' did not contain a loop with signature matching types (<class 'numpy.dtype[bool_]'>, <class 'numpy.dtype[bool_]'>, <class 'numpy.dtyp...
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_pr00 - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_rx - numpy.core._exceptions._UFuncNoLoopError: ufunc 'rx' did not contain a loop with signature matching types (<class 'numpy.dtype[bool_]'>, <class 'numpy.dtype[bool_]'>) -> <class 'numpy.dty...
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_ry - numpy.core._exceptions._UFuncNoLoopError: ufunc 'ry' did not contain a loop with signature matching types (<class 'numpy.dtype[bool_]'>, <class 'numpy.dtype[bool_]'>) -> <class 'numpy.dty...
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_rz - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_sxp - ValueError: Invalid data-type for array
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_tporv - numpy.core._exceptions._UFuncNoLoopError: ufunc 'tporv' did not contain a loop with signature matching types (<class 'numpy.dtype[bool_]'>, <class 'numpy.dtype[bool_]'>, <class 'numpy.dty...
FAILED .pyenv/versions/erfa/lib/python3.11/site-packages/erfa/tests/test_ufunc.py::test_tpstv - numpy.core._exceptions._UFuncNoLoopError: ufunc 'tpstv' did not contain a loop with signature matching types (<class 'numpy.dtype[bool_]'>, <class 'numpy.dtype[bool_]'>, <class 'numpy.dty...
========================================================================== 37 failed, 240 passed, 1 skipped in 2.56s ===========================================================================
saimn commented 1 year ago

Something seems broken in the ufunc logic, scalar work but not numpy arrays:

In [1]: import numpy as np
   ...: import erfa

In [2]: jd = np.linspace(2456855.5, 2456855.5+1.0/24.0/60.0, 60*2+1)

In [3]: erfa.atco13(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, jd[0],
   ...:     0.0, 0.0, 0.0, np.pi/4.0, 0.0, 0.0, 0.0, 1014.0, 0.0, 0.0, 0.5)
Out[3]: 
(1.8822287856305961,
 1.2705653897567488,
 -1.1416413766713571,
 0.002139101854684246,
 6.282590269689792,
 -0.0032873592390111742)

In [4]: erfa.atco13(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, jd,
   ...:     0.0, 0.0, 0.0, np.pi/4.0, 0.0, 0.0, 0.0, 1014.0, 0.0, 0.0, 0.5)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 erfa.atco13(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, jd,
      2     0.0, 0.0, 0.0, np.pi/4.0, 0.0, 0.0, 0.0, 1014.0, 0.0, 0.0, 0.5)

File ~/dev/pyerfa/erfa/core.py:3406, in atco13(rc, dc, pr, pd, px, rv, utc1, utc2, dut1, elong, phi, hm, xp, yp, phpa, tc, rh, wl)
   3218 def atco13(rc, dc, pr, pd, px, rv, utc1, utc2, dut1, elong, phi, hm, xp, yp, phpa, tc, rh, wl):
   3219     """
   3220     ICRS RA,Dec to observed place.  The caller supplies UTC, site
   3221     coordinates, ambient air conditions and observing wavelength.
   (...)
   3404 
   3405     """
-> 3406     aob, zob, hob, dob, rob, eo, c_retval = ufunc.atco13(
   3407         rc, dc, pr, pd, px, rv, utc1, utc2, dut1, elong, phi, hm, xp, yp, phpa, tc, rh, wl)
   3408     check_errwarn(c_retval, 'atco13')
   3409     return aob, zob, hob, dob, rob, eo

ValueError: Invalid data-type for array
avalentino commented 1 year ago

I confirm that I cannot reproduce the issue. @saimn did you try to reproduce the issue in an environment different from pyenv?

saimn commented 1 year ago

I got the tests passing with a miniconda env, which lead me to the reason: this is caused by using -O3, all is fine with -O2.

mhvk commented 1 year ago

Yikes, thanks for all the sleuthing! As noted in #104, I'm definitely happy to put the solution in. But also should check to what extent our own type resolver is still necessary - numpy has fixed some of the problems that forced me to write it...