combine1fiber memory issue

[igoldste]

The combine1fiber function is failing with a MemoryError when combining the spectra of two fibers:

import numpy as np
import pyfits as pf 
from pydl.pydlspec2d.spec2d import combine1fiber

#in this case, fiberFile1='spec-3683-55178-0016.fits' and fiberFile2='spec-3683-55178-0024.fits'

fluxArray=pf.open(fiberFile1)[1].data.field('flux')
loglamArray=pf.open(fiberFile1)[1].data.field('loglam')
ivarArray = pf.open(fiberFile1)[1].data.field('ivar')

nextFlux = pf.open(fiberFile2)[1].data.field('flux')
nextLoglam = pf.open(fiberFile2)[1].data.field('loglam')
nextIvar = pf.open(fiberFile2)[1].data.field('ivar')

newloglam=3.775+np.arange(4000)*1e-4
inloglam=np.array([loglamArray,nextLoglam])
objflux=np.array([fluxArray, nextFlux])
objivar=np.array([ivarArray, nextIvar])

fluxArray, ivarArray = combine1fiber(inloglam=inloglam, objflux=objflux, newloglam=newloglam, objivar=objivar, verbose=True)

---------------------------------------------------------------------------------------------------------------------------------

Traceback (most recent call last):
  File "plt_plate.py", line 53, in <module>
    fluxArray, ivarArray = combine1fiber(inloglam=loglams, objflux=fluxes, newloglam=loglamArray, objivar=ivars, verbose=True)
  File "/global/common/edison/contrib/desi/edison/hpcports_gnu-8.2/pydl-0.3.0_f30e8006-8.2/lib/python2.7/site-packages/pydl-0.3.0-py2.7.egg/pydl/pydlspec2d/spec2d/co
mbine1fiber.py", line 105, in combine1fiber
    wavesort = inloglam[isort]
MemoryError
srun: error: nid00076: task 0: Exited with exit code 1
srun: Terminating job step 881290.0

inloglam shape = (2, 4620) objfluxes shape = (2, 4620) objvar shape = (2, 4620)

I didn't think the arrays involved were large enough to cause memory issues, but I can't get the function to coadd any two fiber spectra. Any help would be appreciated! Thank you.

[weaverba137]

Are you able to reproduce this on any other system, or do you only see this issue at NERSC? If only at NERSC, do you see this only on edison, or on cori as well?

You may also want to try this with a more recent version of pydl.

[igoldste]

It's not just on NERSC- I get the same issue locally with pydl-0.5.0.

[weaverba137]

OK, can you tell me a bit more about what you're trying to do with these spectra?

[igoldste]

I'm trying to combine the spectra of sky fibers from the same plate and MJD that have already gone through sky subtraction.

[weaverba137]

Could you possibly test on the current master branch? If successful, please try with all sky fibers.

[weaverba137]

I haven't heard back from you about testing this, but my own tests suggests that the fix is working, so I'm closing this.