taylorbell57
commented
3 weeks ago @lucasastro, we have a built-in warning that is included in your copy-pasted log that is trying to tell you what to do:
WARNING: NIRSpec GRISM spectra is significantly curved and will very likely benefit from setting meta.curvature to "correct".Because every instrument on JWST is unique, there are different important ECF settings for each instrument. To try to help users get started, we have custom ECF templates for most observing modes in the demos/JWST folder. You'll want to look at the S3_nirspec_fs_template.ecf template in particular. And in general, I strongly recommend you first look closely through all the logs and plots output by Eureka! to try to troubleshoot things on your own.
As for how long it took you to complete Stage 2, I suspect you had the skip_extract_1d step turned on which isn't really needed - it doesn't do anything bad to the data or anything, but we don't use the outputs from that step so it just ends up being a large waste of time.
And then for your question about running Stage 3 without having to re-run Stage 2, this is trivial - you just need to comment out the Stage 2 call in your run_eureka.py file, and remove the s2_meta=s2_meta part from the Stage 3 call.
lucasastro
Instrument
No response
What happened?
I have successfully installed and run Eureka! on the (Wasp-39) test dataset. Now I am running it on a different (NIRSpec G395H) dataset from the JWST MAST archive. Stage 2 was run successfully (in 100+ minutes). Entering Stage 3 produces this error (see output).
I suppose I can try modifying the parameters in the S3**.ecf file (any suggestions which ones? did it stall in the middle of background subtraction?)
However, I'd like to start the pipeline from Stage 3 onwards (assuming the Stage 2 output is legit) rather than having to wait 100+ minutes for every bugfix. Is this possible? It was not straightforward from the run_eureka.py file...
Error traceback output
... (Stage 2 output left out)
Saving Metadata
Total time (min): 104.98
Starting Stage 3 Reduction
Input directory: /Users/lucas/Documents/ScienceHD/REVEAL_HD/JWSTdata/K2-18b/Stage3/S2_2024-06-13_K2-18_run1/ Output directory: /Users/lucas/Documents/ScienceHD/REVEAL_HD/JWSTdata/K2-18b/Stage3/S3_2024-06-13_K2-18_run1/ap6_bg7/ Using ap=6, bg=7, expand=1 Copying S3 control file
Found 4 data file(s) ending in calints.fits Starting file 1 of 4 Reading file 1... Masking NaNs/infs in data arrays... FLUX has 39912 NaNs/infs, which is 0.46% of all pixels. ERR has 39912 NaNs/infs, which is 0.46% of all pixels. V0 has 39912 NaNs/infs, which is 0.46% of all pixels. Locating source position... Source position on detector is row 22. Automatically getting reference files to convert units to electrons... 2024-06-13 13:57:33,353 - CRDS - INFO - Fetching /Users/lucas/crds_cache/references/jwst/nirspec/jwst_nirspec_gain_0025.fits 2.1 M bytes (1 / 1 files) (0 / 2.1 M bytes) Converting from data numbers per second (DN/s) to electrons... Computing clean median frame... WARNING: NIRSpec GRISM spectra is significantly curved and will very likely benefit from setting meta.curvature to "correct". Performing background outlier rejection... Performing CxC background subtraction... 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 950/950 [00:16<00:00, 57.18it/s] Creating figures for background subtraction... 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00, 1.53it/s] Recording y position and width for all integrations... 0%| | 0/950 [00:00<?, ?it/s] Traceback (most recent call last): File "/Users/lucas/Documents/ScienceHD/REVEAL_HD/JWSTdata/K2-18b/run_eureka.py", line 26, in
s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path,
File "/opt/miniconda3/envs/eureka/lib/python3.9/site-packages/eureka/S3_data_reduction/s3_reduce.py", line 447, in reduce
source_pos.source_pos_wrapper(data, meta, log,
File "/opt/miniconda3/envs/eureka/lib/python3.9/site-packages/eureka/S3_data_reduction/source_pos.py", line 73, in source_pos_wrapper
writePos(source_pos(flux[n], meta, data.attrs['shdr'],
File "/opt/miniconda3/envs/eureka/lib/python3.9/site-packages/eureka/S3_data_reduction/source_pos.py", line 162, in source_pos
src_ypos, src_ywidth = source_pos_gauss(flux, meta, m, n, plot)
File "/opt/miniconda3/envs/eureka/lib/python3.9/site-packages/eureka/S3_data_reduction/source_pos.py", line 405, in source_pos_gauss
popt, pcov = curve_fit(gauss, y_pixels, med_row, p0)
File "/opt/miniconda3/envs/eureka/lib/python3.9/site-packages/scipy/optimize/_minpack_py.py", line 839, in curve_fit
raise RuntimeError("Optimal parameters not found: " + errmsg)
RuntimeError: Optimal parameters not found: Number of calls to function has reached maxfev = 1000.
What operating system are you using?
Mac OSX Sonoma 14.4.1
What version of Python are you running?
Python 3.9.7
What Python packages do you have installed?
conda list.txt
Code of Conduct