Wavelength.py passing masked array to pyfits

[followthesheep]

I think there is a bug in Wavelength.py where it is passing a masked array as the value of a header into pyfits. This causes the following error:

/group/ureka/Ureka/python/lib/python2.7/site-packages/pyfits-3.2.4-py2.7-macosx-10.6-x86_64.egg/pyfits/card.pyc in value(self, value)
    562                                   Undefined, np.floating, np.integer,
    563                                   np.complexfloating, np.bool_)):
--> 564             raise ValueError('Illegal value: %r.' % value)
    565 
    566         if isinstance(value, float) and (np.isnan(value) or np.isinf(value)):

ValueError: Illegal value: masked_array(data = 2.1640625,
             mask = False,
       fill_value = 1e+20)

Earlier in Wavelength.py:

    dlam = 0
    central_line = 1024
    step = 0
    while dlam==0:
        line = central_line+(10*step)
        dlam = np.ma.median(np.diff(lams[line,:]))
        if dlam==0:
            line = central_line-(10*step)
            dlam = np.ma.median(np.diff(lams[line,:]))

Somehow, dlam is also a masked array. When this is passed to pyfits to fill the headers, it is crashing.

Has this happened in previous versions?

[KeckDRP]

Tuan this was happening to me too when I tried to modify nmpfit as you suggested.

I believe this goes away if you go back to using the standard version of nmpfit

If you look at the .npy file that you got as an output from the wavelength fitting, you will find masked arrays in there, but they are not there with the old version of bumpy

Luca

On Jun 30, 2015, at 5:21 PM, Tuan Do notifications@github.com wrote:

I think there is a bug in Wavelength.py where it is passing a masked array as the value of a header into pyfits. This causes the following error:

/group/ureka/Ureka/python/lib/python2.7/site-packages/pyfits-3.2.4-py2.7-macosx-10.6-x8664.egg/pyfits/card.pyc in value(self, value) 562 Undefined, np.floating, np.integer, 563 np.complexfloating, np.bool)): --> 564 raise ValueError('Illegal value: %r.' % value) 565 566 if isinstance(value, float) and (np.isnan(value) or np.isinf(value)):

ValueError: Illegal value: masked_array(data = 2.1640625, mask = False, fill_value = 1e+20) Earlier in Wavelength.py:

dlam = 0
central_line = 1024
step = 0
while dlam==0:
    line = central_line+(10*step)
    dlam = np.ma.median(np.diff(lams[line,:]))
    if dlam==0:
        line = central_line-(10*step)
        dlam = np.ma.median(np.diff(lams[line,:]))

Somehow, dlam is also a masked array. When this is passed to pyfits to fill the headers, it is crashing.

Has this happened in previous versions?

— Reply to this email directly or view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/27.

[followthesheep]

Is this related to how nmpfit works, or how numpy works?

[KeckDRP]

Good question. I am not sure. The details must have something to do with the fact that the old numpy used to create copy of the vector when “diagonal” was used, now it creates a view on the vector. Probably when you save that vector, now, you get a masked view if you performed a selection, instead of a subset of the original vector.

We might be able to work around this by converting the masked array into a real array before saving it, but honestly I have no idea how to do that.

On Jun 30, 2015, at 5:26 PM, Tuan Do notifications@github.com wrote:

Is this related to how nmpfit works, or how numpy works?

— Reply to this email directly or view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/27#issuecomment-117424498.

[followthesheep]

I think the masked arrays have .data and .item to get the internal numpy data out. But, the problem will be backwards compatibility. If it wasn't returning masked arrays before, then this change out break it.

[KeckDRP]

Yes I think you are right and indeed no it wasn't returning a masked array before.

We can build logic into it of course: if this is a masked array then extract the data otherwise just use the data directly.

Of course the best solution is still to remove our dependencies on space telescope software but that will take time.

On Jun 30, 2015, at 7:51 PM, Tuan Do notifications@github.com wrote:

I think the masked arrays have .data and .item to get the internal numpy data out. But, the problem will be backwards compatibility. If it wasn't returning masked arrays before, then this change out break it.

— Reply to this email directly or view it on GitHub.

[followthesheep]

I've now added in Wavelength.py a check for masked arrays in adding to the header and will convert to a regular float for the value of dlam.

With this change and the new nmpfit_mos.py I have been able to get the entire pipeline to run with the current version of Ureka. It might be slower now, though it is hard to tell.

[lucarizzi]

Done