search

desihub / gpu_specter

Scratch work for porting spectroperfectionism extractions to GPUs
BSD 3-Clause "New" or "Revised" License
2 stars 3 forks source link

Use cupyx.lapack.posv and fix failing unit tests #66

Closed dmargala closed 3 years ago

dmargala commented 3 years ago

This PR updates the gpu_specter.linalg.cholesky_solve function to use the posv implementation in CuPy (added in v9.0.0). Originally, I included an implementation of posv in gpu_specter while waiting for my CuPy PR to be merged and included in the next major release.

The full suite of unit tests were failing prior to this PR. There are two issues that I fixed so all tests are passing after this PR:

  1. specter comparison with pull value greater than 0.05 sigma (originally reported in #60). I resolved this by excluding masked pixels from the comparison. We had previously discussed ignoring masked pixels in the comparison but that had not made it into the unit tests until now.

  2. Minor api change in cupy v8 -> v9 which I resolved by changing a reference from cp.core.core.ndarray to cp.ndarray.

Before PR:

> pytest py/gpu_specter
======================================================================================= test session starts =======================================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /global/u2/d/dmargala/denali-p8r-pipeline/gpu_specter
collected 32 items                                                                                                                                                                                

py/gpu_specter/test/test_core.py FF.F                                                                                                                                                       [ 12%]
py/gpu_specter/test/test_extract.py ...........                                                                                                                                             [ 46%]
py/gpu_specter/test/test_linalg.py ....                                                                                                                                                     [ 59%]
py/gpu_specter/test/test_polynomial.py ...                                                                                                                                                  [ 68%]
py/gpu_specter/test/test_projection_matrix.py ...                                                                                                                                           [ 78%]
py/gpu_specter/test/test_psfcoeff.py ....                                                                                                                                                   [ 90%]
py/gpu_specter/test/test_spots.py ...                                                                                                                                                       [100%]

============================================================================================ FAILURES =============================================================================================
____________________________________________________________________________________ TestCore.test_compare_gpu ____________________________________________________________________________________

self = <gpu_specter.test.test_core.TestCore testMethod=test_compare_gpu>
...
py/gpu_specter/test/test_core.py:199: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
py/gpu_specter/core.py:594: in extract_frame
    bundle = extract_bundle(
py/gpu_specter/core.py:344: in extract_bundle
    bundle = tuple(cp.asnumpy(x) if isinstance(x, cp.core.core.ndarray) else x for x in bundle)
py/gpu_specter/core.py:344: in <genexpr>
    bundle = tuple(cp.asnumpy(x) if isinstance(x, cp.core.core.ndarray) else x for x in bundle)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'core'

    def __getattr__(name):
        value = _deprecated_attrs.get(name)
        if value is None:
>           raise AttributeError(
                f"module 'cupy' has no attribute {name!r}")
E           AttributeError: module 'cupy' has no attribute 'core'

/global/homes/d/dmargala/.conda/envs/test-gpu-specter/lib/python3.8/site-packages/cupy/__init__.py:874: AttributeError
__________________________________________________________________________________ TestCore.test_compare_specter __________________________________________________________________________________

self = <gpu_specter.test.test_core.TestCore testMethod=test_compare_specter>

    @unittest.skipIf(not specter_available, 'specter not available')
    def test_compare_specter(self):
        if self.comm is not None:
            self.comm.barrier()

        bundlesize = 10
        wavelength = '5760.0,7620.0,0.8'

        specmin = 0
        nspec = 10
        nwavestep = 50
        nsubbundles = 2

        frame_spex = extract_frame(
            self.imgdata, self.psfdata, bundlesize,
            specmin, nspec,
            wavelength=wavelength,
            nwavestep=nwavestep, nsubbundles=nsubbundles,
            comm=self.comm,
            gpu=None,
            loglevel='WARN',
        )

        if self.rank > 0:
            return

        self.assertEqual(frame_spex['specflux'].shape[0], nspec)

        psf = specter.psf.load_psf(self.psffile)

        wavelengths = frame_spex['wave']

        frame_specter = specter.extract.ex2d(
            self.imgdata['image'], self.imgdata['ivar'], psf,
            specmin, nspec, wavelengths,
            xyrange=None, regularize=0.0, ndecorr=False,
            bundlesize=bundlesize, nsubbundles=nsubbundles,
            wavesize=nwavestep,
            full_output=True, verbose=False,
            debug=False, psferr=None,
        )

        self.assertEqual(frame_spex['specflux'].shape, frame_specter['flux'].shape)

        diff = frame_spex['specflux'] - frame_specter['flux']
        norm = np.sqrt(1.0/frame_spex['specivar'] + 1.0/frame_specter['ivar'])
        pull = (diff/norm).ravel()

        #- Require that >99% of the pull values are consistent to
        #- better than 0.01*sigma
        pull_threshold = 0.01
        pull_fraction = np.average(np.abs(pull) < pull_threshold)
        self.assertGreaterEqual(pull_fraction, 0.99)

        #- require that the largest deviation is within 5% of a sigma
>       self.assertLess(np.max(np.abs(pull)), 0.05)
E       AssertionError: 0.06892567423473432 not less than 0.05

py/gpu_specter/test/test_core.py:174: AssertionError
________________________________________________________________________________ TestCore.test_gpu_batch_subbundle ________________________________________________________________________________

self = <gpu_specter.test.test_core.TestCore testMethod=test_gpu_batch_subbundle>
...
py/gpu_specter/test/test_core.py:241: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
py/gpu_specter/core.py:594: in extract_frame
    bundle = extract_bundle(
py/gpu_specter/core.py:344: in extract_bundle
    bundle = tuple(cp.asnumpy(x) if isinstance(x, cp.core.core.ndarray) else x for x in bundle)
py/gpu_specter/core.py:344: in <genexpr>
    bundle = tuple(cp.asnumpy(x) if isinstance(x, cp.core.core.ndarray) else x for x in bundle)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'core'

    def __getattr__(name):
        value = _deprecated_attrs.get(name)
        if value is None:
>           raise AttributeError(
                f"module 'cupy' has no attribute {name!r}")
E           AttributeError: module 'cupy' has no attribute 'core'

/global/homes/d/dmargala/.conda/envs/test-gpu-specter/lib/python3.8/site-packages/cupy/__init__.py:874: AttributeError
===================================================================================== short test summary info =====================================================================================
FAILED py/gpu_specter/test/test_core.py::TestCore::test_compare_gpu - AttributeError: module 'cupy' has no attribute 'core'
FAILED py/gpu_specter/test/test_core.py::TestCore::test_compare_specter - AssertionError: 0.06892567423473432 not less than 0.05
FAILED py/gpu_specter/test/test_core.py::TestCore::test_gpu_batch_subbundle - AttributeError: module 'cupy' has no attribute 'core'
============================================================================= 3 failed, 29 passed in 67.23s (0:01:07) =============================================================================

After PR:

> pytest py/gpu_specter
======================================================================================= test session starts =======================================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /global/u2/d/dmargala/denali-p8r-pipeline/gpu_specter
collected 32 items                                                                                                                                                                                

py/gpu_specter/test/test_core.py ....                                                                                                                                                       [ 12%]
py/gpu_specter/test/test_extract.py ...........                                                                                                                                             [ 46%]
py/gpu_specter/test/test_linalg.py ....                                                                                                                                                     [ 59%]
py/gpu_specter/test/test_polynomial.py ...                                                                                                                                                  [ 68%]
py/gpu_specter/test/test_projection_matrix.py ...                                                                                                                                           [ 78%]
py/gpu_specter/test/test_psfcoeff.py ....                                                                                                                                                   [ 90%]
py/gpu_specter/test/test_spots.py ...                                                                                                                                                       [100%]

================================================================================== 32 passed in 73.24s (0:01:13) ==================================================================================

Notes for running full test suite on Cori GPU:

#- From top-level of gpu_specter on Cori @ NERSC

#- start interactive session on Cori GPU
module purge
module load cgpu
salloc -C gpu -N 1 -G 1 -c 10 -t 60

#- set up env 
module load python cuda/11.3.0
conda create -n test-gpu-specter -y python=3.8 pip pytest numpy scipy numba astropy
source activate test-gpu-specter
pip install fitsio
pip install git+https://github.com/desihub/specter.git@0.10.0#egg=specter
pip install cupy-cuda113 -f https://github.com/cupy/cupy/releases/v9.2.0

#- run full test suite
pytest py/gpu_specter

#- clean up env
conda deactivate
conda env remove -n test-gpu-specter

#- Exit interactive session
exit