spacetelescope / romancal

Python library to process science observations from the Nancy Grace Roman Space Telescope
https://roman-pipeline.readthedocs.io/en/latest/
Other
28 stars 28 forks source link

Bitflags in resample fail with devdeps #1279

Open braingram opened 2 weeks ago

braingram commented 2 weeks ago

~Numpy 2.0~ EDIT: devdeps changes are causing failures for the build_mask function in resample:

dqarr = array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=uint32)
bitvalue = 73728

    def build_mask(dqarr, bitvalue):
        """
        Build a bit mask from an input DQ array and a bitvalue flag.

        Parameters
        ----------
        dqarr : numpy.ndarray
            Input DQ array.
        bitvalue : str
            Bitvalue flag.

        Returns
        -------
        ndarray
            Bit mask where 1 represents good and 0 represents bad.

        Notes
        -----
        - The function interprets the bitvalue flag using the
          `astropy.nddata.bitmask.interpret_bit_flags` function.
        - If the bitvalue is None, the function returns a bit mask with all elements
          set to 1.
        - Otherwise, the function performs a bitwise AND operation between the dqarr and
          the complement of the bitvalue, and then applies a logical NOT operation to
          obtain the bit mask.
        - The resulting bit mask is returned as a `numpy.ndarray` of dtype `numpy.uint8`.
        """
        bitvalue = interpret_bit_flags(
            bitvalue, flag_name_map={dq.name: dq.value for dq in pixel}
        )

        if bitvalue is None:
            return np.ones(dqarr.shape, dtype=np.uint8)
>       return np.logical_not(np.bitwise_and(dqarr, ~bitvalue)).astype(np.uint8)
E       OverflowError: Python integer -73729 out of bounds for uint32

Full run (which also show webbpsf errors): https://github.com/spacetelescope/romancal/actions/runs/9569032524/job/26380655507?pr=1278#step:10:2228

This looks to be due to the use of a bitwise not on a python number (instead of a numpy scalar). This produces a negative number and an error with numpy 2.

>> ~1
-2

With numpy < 2:

>> np.bitwise_and(np.uint32(2), ~1)
2

With numpy 2:

>> np.bitwise_and(np.uint32(1), ~1)
OverflowError: Python integer -2 out of bounds for uint32
schlafly commented 2 weeks ago

Probably need to do: bitvalue = np.array(bitvalue).astype(dqarr.dtype)

braingram commented 6 days ago

This doesn't appear to be entirely a numpy issue as the devdeps are failing with a similar error: https://github.com/spacetelescope/romancal/actions/runs/9662228750/job/26651704014#step:10:1924

>       return np.logical_not(np.bitwise_and(dqarr, ~bitvalue)).astype(np.uint8)
E       OverflowError: Python integer -1 out of bounds for uint32

when numpy 1.26 is installed. https://github.com/spacetelescope/romancal/actions/runs/9662228750/job/26651704014#step:10:583