inspect_extracted_spectrum() zero-size array to reduction operation fmin which has no identity

[rjs3273]

As mentioned in another discussion, I have been getting at error in ap_extract() in version 0.5.0.

It's not blocking me so i) I have not investigated the src code in detail to make a useful suggestion of what may be the problem ii) it is not urgent iii) I have not yet done any of the basics like testing reproducibility on different data.

I get the same error with either inspect_extracted_spectrum() or ap_extract(display=True)

ValueError                                Traceback (most recent call last)
/var/folders/2s/36cb8cyd6p98wg_9d5jh9vrm0000gp/T/ipykernel_76336/1823149230.py in <module>
      1 target2D.ap_extract(apwidth=13,optimal=True,skysep=10,skywidth=10,skydeg=1,display=False)
----> 2 target2D.inspect_extracted_spectrum()

~/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/twodspec.py in inspect_extracted_spectrum(self, spec_id, display, renderer, width, height, return_jsonstring, save_fig, fig_type, filename, open_iframe)
   4233                         min(
   4234                             np.nanmin(
-> 4235                                 sigma_clip(count, sigma=5.0, masked=False)
   4236                             ),
   4237                             np.nanmin(

<__array_function__ internals> in nanmin(*args, **kwargs)

~/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/numpy/lib/nanfunctions.py in nanmin(a, axis, out, keepdims)
    317         # Fast, but not safe for subclasses of ndarray, or object arrays,
    318         # which do not implement isnan (gh-9009), or fmin correctly (gh-8975)
--> 319         res = np.fmin.reduce(a, axis=axis, out=out, **kwargs)
    320         if np.isnan(res).any():
    321             warnings.warn("All-NaN slice encountered", RuntimeWarning,

ValueError: zero-size array to reduction operation fmin which has no identity

[cylammarco]

Hi @rjs3273, I think I fixed that in one of the very recent commits. Can you please try to pull and see if the error persists and let me know? Thanks!

[rjs3273]

Yes and No. I am still getting the error, but I have now done what I said I should have done earlier and tried a few other files. I tested several spectra taken on the same night and only one of them gives this error, so it must be something funny about that particular FITS file. I will try to figure out what is wrong with it. It looks fine by eye.

One possible clue is that the file that gives this error also gives hundreds of "invalid value" warnings from ap_extract(). I guess there must be some bad values in the FITS. I will investigate.

target2D.ap_extract(apwidth=13,optimal=True,skysep=10,skywidth=10,skydeg=1,display=False)

/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/twodspec.py:3481: RuntimeWarning:
invalid value encountered in true_divide
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:221: RuntimeWarning:
invalid value encountered in double_scalars
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:224: RuntimeWarning:
invalid value encountered in double_scalars
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/twodspec.py:3481: RuntimeWarning:
invalid value encountered in true_divide
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:221: RuntimeWarning:
invalid value encountered in double_scalars
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:224: RuntimeWarning:
invalid value encountered in double_scalars
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/twodspec.py:3481: RuntimeWarning:
invalid value encountered in true_divide
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:221: RuntimeWarning:
invalid value encountered in double_scalars
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:224: RuntimeWarning:
invalid value encountered in double_scalars
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/twodspec.py:3481: RuntimeWarning:
invalid value encountered in true_divide
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:221: RuntimeWarning:
invalid value encountered in double_scalars
/Users/rjs/miniconda3/envs/astroconda_mookodi/lib/python3.7/site-packages/aspired/extraction.py:224: RuntimeWarning:
invalid value encountered in double_scalars

There are hundreds and hundreds of them before the plotting then fails.

[rjs3273]

I had a good rummage around in that one particular FITS and could not find anything especially odd about it. Seems like a valid spectrum to me.

I looked in the code and the warnings I am getting come from optimal_extraction_horne86(). At line 184 in extraction.py

# step 5 - construct the spatial profile
P = profile

P is an array of all zeroes. That then propagates through and causes the numerical errors later. I think the output array is then empty causing the plotting error. I currently however have no idea why for this file alone, the profile is coming out all zeroes. Every other FITS I have tried through the same scripts gives a valid profile.

I then swapped to ap_extract(optimal=True,algorithm='marsh89') and no longer get any errors. It all works.

So there is something in this one particular spectrum FITS that is making it fail on the Horne algorithm. I am happy just carrying on with the Marsh algorithm for now and not worrying about this. If you want the bad FITS that triggered the error, I can provide it.

[cylammarco]

I suspect it is related to the trace. I have updated the dev to check if the profile is all zeros, if that happens, the profile will be set to ones. However, that forces the extraction to be tophat, the profile shouldn't be all zeros in any case.

If you could share the data with me, I can look into it further. Thanks!

[rjs3273]

Thanks, Marco. I sent you the FITS via other routes. I didn't think attaching it in here was helpful. As I said before, this is not blocking anything because I can use Marsh89, but it does seem worth at least a quick look to see if the problem is obvious to you.

[rjs3273]

Marco is unable to replicate the error with the same file and commands.

Maybe I am not installing properly?

I am using pip install git+https://github.com/cylammarco/ASPIRED.git@dev-0.5.X --upgrade

and the install is saying Resolved https://github.com/cylammarco/ASPIRED.git to commit b8e9704c89f6c99f98cd807a47721a3294af32bf

Is that the correct commit?

[cylammarco]

Please try uninstalling and installing it again without using cache. That message you showed is pointing to an older commit. Probably because I am not assigning version number properly in dev :/

[rjs3273]

Thanks, Marco. I'll have to go and read the pip manual. :)

[rjs3273]

I was running in a conda environment so the quickest and easiest was just delete that environment and start a new one from scratch. As you expected, it all works now, so it does look like something somewhere being cached. This issue can be closed?