data gap masking

[HajimeKawahara]

when analyzing the old IRD data (Jupiter), we find that the data gap needs masking.

Sample Code: https://github.com/HajimeKawahara/jovispec/tree/develop

• analysis code: analysis/irdjupiter.py
• plotting: analysis/read_spectrum.py

[YuiKasagi]

By setting the additional mask in your code (irdjupiter.py), for example,

gap_mask = np.zeros(median_image.shape,dtype=bool)
gap_mask[63::128,:] = True
gap_mask[64::128,:] = True

hotpix_mask = hotpix_mask | gap_mask

I could get the masked (interpolated) spectrum. (Wavelength calibration is worse due to a few number of ThAr emissions...)

To apply gap_mask, I slightly modified the code. #87 Does this match with what you want to do?

[HajimeKawahara]

Sorry for the late reply, but I couldn't reproduce the results. This ends up the following error:

Traceback (most recent call last):
  File "/home/kawahara/jovispec/analysis/irdjupiter.py", line 46, in <module>
    thar.calibrate_wavelength()
  File "/home/kawahara/pyird/src/pyird/utils/irdstream.py", line 568, in calibrate_wavelength
    wavsol, data = wavcal_thar(
  File "/home/kawahara/pyird/src/pyird/spec/wavcal.py", line 232, in wavcal_thar
    pix_dat = np.where(dat[i, :] == max(dat[i, pix_med]))[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

Can you check my code? https://github.com/HajimeKawahara/jovispec/blob/develop/analysis/irdjupiter.py

[YuiKasagi]

I could not confirm the error, but I think that the iteration process removed too many ThAr lines. I will further check this issue and revise the code if possible. As a workaround for now, would it work if you set the option maxiter, for example, wavecal_thar(maxiter=10)?

[HajimeKawahara]

shirochan:~/jovispec/analysis(develop)>python irdjupiter.py
fitsid: [8140833, 8140945, 8141330]
median combine:  
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1317.58it/s]
/home/kawahara/pyird/src/pyird/image/aptrace.py:236: UserWarning: Looks a single fiber aperture on the detector.
  warnings.warn("Looks a single fiber aperture on the detector.", UserWarning)
cross-section: row  581
100%|███████████████████████████████████████████████████████████████████████████████████| 21/21 [00:10<00:00,  2.09it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 61.23it/s]
fitsid: [8133836]
median combine:  
100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1121.77it/s]
hotpix mask = 0.55 percent
Skipped 3 files because they already exists.
0it [00:00, ?it/s]
0it [00:00, ?it/s]
continuum is fitted with order_fit = 23.
median combine:  _cp
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1073.99it/s]
/home/kawahara/pyird/src/pyird/utils/irdstream.py:512: RuntimeWarning: All-NaN slice encountered
  median_image = np.nanmedian(imall, axis=0)
100%|███████████████████████████████████████████████████████████████████████████████████| 21/21 [00:11<00:00,  1.86it/s]
fitsid: [8143952, 8144142, 8144237, 8144415]
Skipped 4 files because they already exists.
0it [00:00, ?it/s]
median combine:  _cp
100%|███████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 1486.95it/s]
/home/kawahara/pyird/src/pyird/utils/irdstream.py:512: RuntimeWarning: All-NaN slice encountered
  median_image = np.nanmedian(imall, axis=0)
100%|███████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 43.53it/s]
H band
/home/kawahara/pyird/src/pyird/spec/wavcal.py:193: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  pdat1 = pd.concat([pdat1, df_tmp], ignore_index=True)
standard deviation of residuals of 1st iteration = 0.02970
/home/kawahara/pyird/src/pyird/spec/wavcal.py:229: RuntimeWarning: All-NaN slice encountered
  if np.nanmax(med[ind_low:ind_upp]) > np.nanpercentile(med[med>0], 80): #CHECK!!
[False False False ... False False False]
/home/kawahara/pyird/src/pyird/spec/wavcal.py:237: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  pdat2 = pd.concat([pdat2, df_tmp], ignore_index=True)
Traceback (most recent call last):
  File "/home/kawahara/jovispec/analysis/irdjupiter.py", line 46, in <module>
    thar.calibrate_wavelength(maxiter=10)
  File "/home/kawahara/pyird/src/pyird/utils/irdstream.py", line 568, in calibrate_wavelength
    wavsol, data = wavcal_thar(
  File "/home/kawahara/pyird/src/pyird/spec/wavcal.py", line 232, in wavcal_thar
    pix_dat = np.where(dat[i, :] == max(dat[i, pix_med]))[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

[HajimeKawahara]

As you told me, np.nanmax solves the issue! Thanks!