Open mithun1996-png opened 1 week ago
jbrande
commented
1 week ago Can you post your Stage 2 and Stage 3 ECFs? Also, what dataset are you trying to analyze? You may have specified some earlier step incorrectly, as every pixel in the image is marked as bad, so there's no actual data being analyzed after those steps.
mithun1996-png
commented
1 week ago Data Specifications (needed to input the MAST JWST Search): Object name : HD 189733b Data type : Time series Instruments : NIRCam Exposure type : NRC_TSGRISM (Spectrum time series) Filter : F322W2, F444W Program (proposal) number : 1633 Principal investigator of the observing program : Deming Dates of observation : 2022-08-25 (F444W), 2022-08-29 (F322W2) Total exposure duration : 21,504.2 s (F444W), 21,383.1 s (F322W2) [optional]
# Eureka! Control File for Stage 2: Data Reduction
# Stage 2 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-2
suffix rateints # Data file suffix
# Controls the cross-dispersion extraction
tsgrism_extract_height None # Use None to rely on the default parameters
# Note: different instruments and modes will use different steps by default
skip_flat_field False
skip_photom True # Recommended to skip to get better uncertainties out of Stage 3. Set this to False if you want to compute calibrated stellar spectra.
skip_extract_1d True
# Diagnostics
hide_plots True # If True, plots will automatically be closed rather than popping up
# Project directory
topdir /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01
# Directories relative to topdir
inputdir /output_stage_1/
outputdir /output_stage_2_new/# Eureka! Control File for Stage 3: Data Reduction
# Stage 3 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-3
ncpu 4 # Number of CPUs
nfiles 1 # The number of data files to analyze simultaneously
max_memory 0.5 # The maximum fraction of memory you want utilized by read-in frames (this will reduce nfiles if need be)
indep_batches False # Independently treat each batch of files? Strongly recommended to leave this as False unless you have a clear reason to set it to True.
suffix calints # Data file suffix
calibrated_spectra False # Set True to generate flux-calibrated spectra/photometry in mJy
# Set False to convert to electrons
# Subarray region of interest
ywindow [5,64] # Vertical axis as seen in DS9
xwindow [780,2000] # Horizontal axis as seen in DS9
src_pos_type gaussian # Determine source position when not given in header (Options: header, gaussian, weighted, max, hst, or a numeric value)
record_ypos True # Option to record the y position and width for each integration (only records if src_pos_type is gaussian)
poly_wavelength True # Use an updated polynomial wavelength solution for NIRCam longwave spectroscopy instead of the linear wavelength solution currently assumed by STScI
dqmask True # Mask pixels with an odd entry in the DQ array
expand 1 # Super-sampling factor along cross-dispersion direction
# Outlier rejection along time axis
ff_outlier True # Set False to use only background region (recommended for deep transits)
# Set True to use full frame (works well for shallow transits/eclipses)
bg_thresh [10,10] # Double-iteration X-sigma threshold for outlier rejection along time axis
# Background parameters
bg_hw 11 # Half-width of exclusion region for BG subtraction (relative to source position)
bg_deg 1 # Polynomial order for column-by-column background subtraction, -1 for median of entire frame
bg_method median # Options: std (Standard Deviation), median (Median Absolute Deviation), mean (Mean Absolute Deviation)
p3thresh 5 # X-sigma threshold for outlier rejection during background subtraction
bg_row_by_row True # Row-by-row BG subtraction (only useful for NIRCam)
bg_x1 0 # Left edge of exclusion region for row-by-row BG subtraction
bg_x2 512 # Right edge of exclusion region for row-by-row BG subtraction
# Spectral extraction parameters
spec_hw 7 # Half-width of aperture region for spectral extraction (relative to source position)
fittype meddata # Method for constructing spatial profile (Options: smooth, meddata, poly, gauss, wavelet, or wavelet2D)
median_thresh 10 # Sigma threshold when flagging outliers in median frame, when fittype=meddata and window_len > 1
window_len 7 # Smoothing window length, for median frame or when fittype = smooth or meddata (when computing median frame). Can set to 1 for no smoothing when computing median frame for fittype=meddata.
prof_deg 3 # Polynomial degree, when fittype = poly
p5thresh 10 # X-sigma threshold for outlier rejection while constructing spatial profile
p7thresh 7 # X-sigma threshold for outlier rejection during optimal spectral extraction
# Curvature treatment
curvature correct # How to manage the curved trace on the detector (Options: None, correct)
# Diagnostics
isplots_S3 3 # Generate few (1), some (3), or many (5) figures (Options: 1 - 5)
nplots 5 # How many of each type of figure do you want to make per file?
vmin 0.97 # Sets the vmin of the color bar for Figure 3101.
vmax 1.03 # Sets the vmax of the color bar for Figure 3101.
time_axis 'y' # Determines whether the time axis in Figure 3101 is along the y-axis ('y') or the x-axis ('x')
testing_S3 False # Boolean, set True to only use last file and generate select figures
hide_plots True # If True, plots will automatically be closed rather than popping up
save_output True # Save outputs for use in S4
save_fluxdata Ture # Save the much larger FluxData.h5 outputs which can be useful for debugging or comparisons between different pipelines
verbose True # If True, more details will be printed about steps
# Project directory
topdir /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01
# Directories relative to topdir
inputdir /output_stage_2/S2_2024-11-05_nircam_wfss_template_run1/
outputdir /stage_3/
kevin218
commented
1 week ago Does Stage 3 work when you don't run row-by-row background subtraction?
taylorbell57
commented
1 week ago Are you running F322W2 and F444W observations simultaneously? You can only do one filter at a time, and based on your previous message it seems to me that you're working on both filters at the same time. You'll need to put the F322W2 and F444W files in separate folders. Also make sure that you're only working on the spectroscopy files and not also the simultaneously collected photometry files
taylorbell57
commented
1 week ago @mithun1996-png, can you confirm whether you're still dealing with this issue?
mithun1996-png
commented
3 days ago this issue is still happen ,but in stage 3 ecf file when i fix row by row background subtraction is true then this issue actually come up. but if i set row by row background subtraction then their is no issue like that . so can you give a suggestion what can do ,what is the reason for that issue .
taylorbell57
commented
3 days ago @mithun1996-png, I looked on MAST and there should only be 11 F444W files and 14 F322W2 files; because your log statements up above state Found 25 data file(s) ending in calints.fits, I am quite sure that you are incorrectly trying to work on F322W2 and F444W observations at the same time. Please separate the F322W2 files from the F444W files and only work on one of them at a time. If you are still encountering an issue when working on only one data set at a time, please copy-paste your new error traceback output
mithun1996-png
commented
2 days ago we get same error when working on only one data set at a time , i am working using F322W filter ecf file for this filter :
# Eureka! Control File for Stage 3: Data Reduction
# Stage 3 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-3
ncpu 4 # Number of CPUs
nfiles 1 # The number of data files to analyze simultaneously
max_memory 0.5 # The maximum fraction of memory you want utilized by read-in frames (this will reduce nfiles if need be)
indep_batches False # Independently treat each batch of files? Strongly recommended to leave this as False unless you have a clear reason to set it to True.
suffix calints # Data file suffix
calibrated_spectra False # Set True to generate flux-calibrated spectra/photometry in mJy
# Set False to convert to electrons
# Subarray region of interest
ywindow [5,64] # Vertical axis as seen in DS9
xwindow [4,1704] # Horizontal axis as seen in DS9
src_pos_type gaussian # Determine source position when not given in header (Options: header, gaussian, weighted, max, hst, or a numeric value)
record_ypos True # Option to record the y position and width for each integration (only records if src_pos_type is gaussian)
poly_wavelength True # Use an updated polynomial wavelength solution for NIRCam longwave spectroscopy instead of the linear wavelength solution currently assumed by STScI
dqmask True # Mask pixels with an odd entry in the DQ array
expand 1 # Super-sampling factor along cross-dispersion direction
# Outlier rejection along time axis
ff_outlier True # Set False to use only background region (recommended for deep transits)
# Set True to use full frame (works well for shallow transits/eclipses)
bg_thresh [10,10] # Double-iteration X-sigma threshold for outlier rejection along time axis
# Background parameters
bg_hw 11 # Half-width of exclusion region for BG subtraction (relative to source position)
bg_deg 1 # Polynomial order for column-by-column background subtraction, -1 for median of entire frame
bg_method median # Options: std (Standard Deviation), median (Median Absolute Deviation), mean (Mean Absolute Deviation)
p3thresh 5 # X-sigma threshold for outlier rejection during background subtraction
bg_row_by_row True # Row-by-row BG subtraction (only useful for NIRCam)
bg_x1 1850 # Left edge of exclusion region for row-by-row BG subtraction
bg_x2 2048 # Right edge of exclusion region for row-by-row BG subtraction
# Spectral extraction parameters
spec_hw 7 # Half-width of aperture region for spectral extraction (relative to source position)
fittype meddata # Method for constructing spatial profile (Options: smooth, meddata, poly, gauss, wavelet, or wavelet2D)
median_thresh 10 # Sigma threshold when flagging outliers in median frame, when fittype=meddata and window_len > 1
window_len 7 # Smoothing window length, for median frame or when fittype = smooth or meddata (when computing median frame). Can set to 1 for no smoothing when computing median frame for fittype=meddata.
prof_deg 3 # Polynomial degree, when fittype = poly
p5thresh 10 # X-sigma threshold for outlier rejection while constructing spatial profile
p7thresh 7 # X-sigma threshold for outlier rejection during optimal spectral extraction
# Curvature treatment
curvature correct # How to manage the curved trace on the detector (Options: None, correct)
# Diagnostics
isplots_S3 3 # Generate few (1), some (3), or many (5) figures (Options: 1 - 5)
nplots 5 # How many of each type of figure do you want to make per file?
vmin 0.97 # Sets the vmin of the color bar for Figure 3101.
vmax 1.03 # Sets the vmax of the color bar for Figure 3101.
time_axis 'y' # Determines whether the time axis in Figure 3101 is along the y-axis ('y') or the x-axis ('x')
testing_S3 False # Boolean, set True to only use last file and generate select figures
hide_plots True # If True, plots will automatically be closed rather than popping up
save_output True # Save outputs for use in S4
save_fluxdata Ture # Save the much larger FluxData.h5 outputs which can be useful for debugging or comparisons between different pipelines
verbose True # If True, more details will be printed about steps
# Project directory
topdir /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01
# Directories relative to topdir
inputdir /output_stage_2_F322W/S2_2024-11-20_nircam_wfss_template_run2/
outputdir /Output_stage_3/filter_F322W/Error outcome working with F322W data
Starting Stage 3 Reduction
Input directory: /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/output_stage_2_F322W/S2_2024-11-20_nircam_wfss_template_run2/
Found 14 data file(s) ending in calints.fits
Output directory: /media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Output_stage_3/filter_F322W/S3_2024-11-21_nircam_wfss_template_run1/ap7_bg11/
Using ap=7, bg=11, expand=1
Copying S3 control file
Starting file 1 of 14
Reading file 1...
Performing RxR background subtraction...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 756/756 [00:05<00:00, 135.27it/s]
Creating figures for background subtraction...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00, 1.32it/s]
Masking NaNs/infs in data arrays...
FLUX has 75826800 NaNs/infs, which is 100.00% of all pixels.
WARNING: Your region of interest may be off the edge of the detector subarray. Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
ERR has 75826800 NaNs/infs, which is 100.00% of all pixels.
WARNING: Your region of interest may be off the edge of the detector subarray. Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
V0 has 75826800 NaNs/infs, which is 100.00% of all pixels.
WARNING: Your region of interest may be off the edge of the detector subarray. Masking NaN/inf regions and continuing, but you should really stop and reconsider your choices.
Locating source position...
Traceback (most recent call last):
File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/demos/JWST/run_eureka.py", line 27, in <module>
spec, meta = s3.reduce(eventlabel, ecf_path=ecf_path)
File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 350, in reduce
source_pos.source_pos_wrapper(data, meta, log, m)
File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 101, in source_pos_wrapper
meta.src_ypos = source_pos(flux[integ], meta, data.attrs['shdr'],
File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 162, in source_pos
src_ypos, src_ywidth = source_pos_gauss(flux, meta, m, n, plot)
File "/media/seps3/abb231cd-8a52-48df-a46a-5caa396b751c/data_anlysis_01/Eureka/src/eureka/S3_data_reduction/source_pos.py", line 410, in source_pos_gauss
p0 = [np.ma.max(med_row), pos_max, sigma0, np.ma.median(med_row)]
File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/ma/core.py", line 6795, in max
return obj.max(axis=axis, fill_value=fill_value, out=out, **kwargs)
File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/ma/core.py", line 5922, in max
result = self.filled(fill_value).max(
File "/home/seps3/miniconda3/envs/eureka/lib/python3.10/site-packages/numpy/core/_methods.py", line 41, in _amax
return umr_maximum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation maximum which has no identityAlright, I just tested Stage 3's NIRCam row-by-row background subtraction, and I can confirm that there does indeed appear to be some bug that results in all pixels becoming masked. While we work to sort this out, I recommend you stick with column-by-column background subtraction only and set bg_row_by_row to False.
On an unrelated note, I saw that you have accidentally set save_fluxdata to Ture rather than True. I also generally recommend leaving save_fluxdata to False unless there is something in particular you want from that file; the FluxData.h5 files are really large and are only really useful for detailed debugging or comparisons between different pipelines
Instrument
NIRCam (Stages 1-3)
What happened?
as i am using NIRCam data , so row by row background subtraction is very important , so when i set rorw _by_ror_bg to True , then this type of error is here .
Error traceback output
What operating system are you using?
linux
What version of Python are you running?
python3.10.14
What Python packages do you have installed?
Code of Conduct