numpy 2 promotion changes

[DougBurke]

So, with NumPy 2.0 we get a number of failures if you say

% export NPY_PROMOTION_STATE=weak_and_warn
% pytest
...

Here is an example (after tweaking validate_replacement_warning to provide some information on the extra warnings:

% pytest sherpa/astro/tests/test_astro_data_swift_unit.py

...

>       assert len(ws) == 1, [str(w.message) for w in ws]
E       AssertionError: ['result dtype changed due to the removal of value-based promotion from NumPy. Changed from int64 to int16.', "The min...herpa-conda/sherpa/../sherpa-test-data/sherpatest/swxpc0to12s6_20130101v014.rmf' was 0 and has been replaced by 1e-10"]
E       assert 2 == 1
E        +  where 2 = len([<warnings.WarningMessage object at 0x7cbfe78af050>, <warnings.WarningMessage object at 0x7cbfe78ace50>])

So we can see that there is some warnings due to type conversion (or the lack thereof) which need reviewing.

[DougBurke]

In some CIAO regression tests *but when run using a numpy 2 build) I'm seeing

/home/dburke/micromamba/envs/ciaot/lib/python3.11/site-packages/sherpa/utils/__init__.py:4072: UserWarning: result dtype changed due to the removal of value-based promotion from NumPy. Changed from float64 to float32.
  tol_act = 2.0 * DBL_EPSILON * abs(xb) + tol / 2.0

from

  4057              warning('%s: %s fa * fb < 0 is not met', __name__, fcn.__name__)
  4058              return [[None, None], [[None, None], [None, None]], myfcn.nfev]
  4059  
  4060          xc = xa
  4061          fc = fa
  4062          DBL_EPSILON = np.finfo(np.float32).eps
  4063          while myfcn.nfev < maxfev:
  4064  
  4065              prev_step = xb - xa
  4066  
  4067              if abs(fc) < abs(fb):
  4068                  xa, fa = xb, fb
  4069                  xb, fb = xc, fc
  4070                  xc, fc = xa, fa
  4071  
  4072              tol_act = 2.0 * DBL_EPSILON * abs(xb) + tol / 2.0
  4073              new_step = (xc - xb) / 2.0
  4074  
  4075              if abs(fb) <= tol:
  4076                  return [[xb, fb], [[xa, fa], [xb, fb]], myfcn.nfev]
  4077  
  4078              if abs(new_step) <= tol_act:
  4079                  if np.sign(fb) != np.sign(fa):

[DougBurke]

Oh joy. A lot of the warnings come from the RMF code where you get warnings like

/home/dburke/micromamba/envs/sherpa-conda/lib/python3.11/site-packages/astropy/io/fits/fitsrec.py:806: UserWarning: result dtype changed due to the removal of value-based promotion from NumPy. Changed from int64 to int32.
  offset = field[idx, 1] + self._heapoffset
/home/dburke/micromamba/envs/sherpa-conda/lib/python3.11/site-packages/astropy/io/fits/fitsrec.py:817: UserWarning: result dtype changed due to the removal of value-based promotion from NumPy. Changed from int64 to int32.
  arr_len = count * dt.itemsize
/home/dburke/sherpa/sherpa-conda/sherpa/astro/io/pyfits_backend.py:890: UserWarning: result dtype changed due to the removal of value-based promotion from NumPy. Changed from int64 to int16.
  numelt += sum(row)
/home/dburke/sherpa/sherpa-conda/sherpa/astro/io/pyfits_backend.py:890: RuntimeWarning: overflow encountered in scalar add
  numelt += sum(row)

This code sees significant changes in

• 1921

so it will be easier to look at those changes once #1921 has been merged.

[DougBurke]

EDITED TO ADD See https://github.com/astropy/astropy/issues/16819

So, some of the messages come from astropy 6.1.2:

>>> import importlib.metadata
>>> importlib.metadata.version("astropy")
'6.1.2'

we can print out the first three rows of each column (some columns are scalars and some are variable-length). It looks like the first variable-length column has the issue (the warning is not necessarily repeated so this could happen to all columns but only reported for the first):

>>> from astropy.io import fits
>>> hdus = fits.open("sherpa-test-data/sherpatest/3c273.rmf")
>>> matrix = hdus["MATRIX"]
>>> matrix.columns
ColDefs(
    name = 'ENERG_LO'; format = 'E'; unit = 'keV'
    name = 'ENERG_HI'; format = 'E'; unit = 'keV'
    name = 'N_GRP'; format = 'I'
    name = 'F_CHAN'; format = 'PI(2)'
    name = 'N_CHAN'; format = 'PI(2)'
    name = 'MATRIX'; format = 'PE(81)'
)
>>> for col in matrix.columns:
...     print(f"### {col.name}")
...     print(matrix.data[col.name][0:3])
...     print("")
... 
### ENERG_LO
[0.1  0.11 0.12]

### ENERG_HI
[0.11 0.12 0.13]

### N_GRP
[1 1 1]

### F_CHAN
/home/dburke/micromamba/envs/sherpa-conda/lib/python3.11/site-packages/astropy/io/fits/fitsrec.py:806: UserWarning: result dtype changed due to the removal of value-based promotion from NumPy. Changed from int64 to int32.
  offset = field[idx, 1] + self._heapoffset
/home/dburke/micromamba/envs/sherpa-conda/lib/python3.11/site-packages/astropy/io/fits/fitsrec.py:817: UserWarning: result dtype changed due to the removal of value-based promotion from NumPy. Changed from int64 to int32.
  arr_len = count * dt.itemsize
[array([8], dtype=int16) array([8], dtype=int16) array([8], dtype=int16)]

### N_CHAN
[array([7], dtype=int16) array([8], dtype=int16) array([9], dtype=int16)]

### MATRIX
[array([5.3483307e-01, 3.1740347e-01, 1.1758100e-01, 2.6072497e-02,
        3.7736967e-03, 3.2021964e-04, 1.6185948e-05], dtype=float32)
 array([4.1695124e-01, 3.4134436e-01, 1.7506562e-01, 5.4360949e-02,
        1.0884368e-02, 1.2962134e-03, 9.2803697e-05, 4.3816003e-06],
       dtype=float32)
 array([2.9488873e-01, 3.3307877e-01, 2.3614250e-01, 1.0234590e-01,
        2.8314942e-02, 4.7199526e-03, 4.7669691e-04, 3.1340638e-05,
        1.1161736e-06], dtype=float32)                             ]

>>>