Closed jovian-explorer closed 2 years ago
taylorbell57
commented
2 years ago Have you looked at the plots produced for Stage 5 from the lsq and emcee fits (and any plots made for the dynesty fit before the code crashes)? Does it look like those fitting methods are actually doing a good job and there's just something glitchy with dynesty?
jovian-explorer
commented
2 years ago These are the plots for lsq
jovian-explorer
commented
2 years ago These are for emcee
jovian-explorer
commented
2 years ago Dynesty doen't produce any plots before crashing. It does generate a couple of files before crashing. I am copy pasting the log file here, hoping it helps sort the issue.
Starting Stage 2 Reduction
Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage1/
Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage2/S2_2022-05-18_wasp39b_run1/
Copying S2 control file
Found 1 data file(s) ending in rateints.fits
Starting file 1 of 1
Editing (in place) the waverange in the input file
Running the Spec2Pipeline
Generating x1dints figure
Total time (min): 0.76
Saving Metadata
Starting Stage 3 Reduction
Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage2/S2_2022-05-18_wasp39b_run1/
Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage3/S3_2022-05-18_wasp39b_run1/ap6_bg7/
Using ap=6, bg=7
Copying S3 control file
Found 1 data file(s) ending in calints.fits
WARNING: Are you using real JWST data? If so, you should edit the flag_bg() function in nirspec.py and look at Issue #193 on Github!
Reading file 1 of 1
Source position on detector is row 13.
Automatically getting reference files to convert units to electrons
Converting from data numbers per second (DN/s) to electrons
Performing background outlier rejection
Performing background subtraction
Creating figures for background subtraction
Performing optimal spectral extraction
Creating figures for optimal spectral extraction
Total time (min): 1.45
Stage 3 MAD = 7572 ppm
Generating figure
Saving Metadata
Starting Stage 4: Generate Light Curves
Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage3/S3_2022-05-18_wasp39b_run1/ap6_bg7/
Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage4/S4_2022-05-18_wasp39b_run1/ap6_bg7/
Copying S4 control file
Loading S3 save file:
/home/keshav/Data/JWST-Sim/NIRSpec/Stage3/S3_2022-05-18_wasp39b_run1/ap6_bg7/S3_wasp39b_ap6_bg7_SpecData.h5
Stage 4 MAD = 5700.39 ppm
Generating light curves
Bandpass 0 = 1.500 - 1.650
Bandpass 1 = 1.650 - 1.800
Bandpass 2 = 1.800 - 1.950
Bandpass 3 = 1.950 - 2.100
Bandpass 4 = 2.100 - 2.250
Bandpass 5 = 2.250 - 2.400
Bandpass 6 = 2.400 - 2.550
Bandpass 7 = 2.550 - 2.700
Bandpass 8 = 2.700 - 2.850
Bandpass 9 = 2.850 - 3.000
Bandpass 10 = 3.000 - 3.150
Bandpass 11 = 3.150 - 3.300
Bandpass 12 = 3.300 - 3.450
Bandpass 13 = 3.450 - 3.600
Bandpass 14 = 3.600 - 3.750
Bandpass 15 = 3.750 - 3.900
Bandpass 16 = 3.900 - 4.050
Bandpass 17 = 4.050 - 4.200
Bandpass 18 = 4.200 - 4.350
Bandpass 19 = 4.350 - 4.500
Total time (min): 0.24
Saving results
Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage4/S4_2022-05-18_wasp39b_run1/ap6_bg7/
Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage5/S5_2022-05-18_wasp39b_run1/ap6_bg7/
Copying S5 control file
Copying S5 parameter control file
Starting Channel 1 of 20
=========================
Starting dynesty fit.
Starting lnprob: -2174602367.8695884
Running dynesty...
Summary
=======
nlive: 1024
niter: 69006
ncall: 2002026
eff(%): 3.498
logz: -6546321.875 +/- 0.312
Number of posterior samples is 70030
Ending lnprob: -6546278.727066302
evamariaa
commented
2 years ago I have seen this error before too and I think it's a convergence problem i.e. the corner package can't deal with dynesty posterior samples where the it can't compute ranges. You can try computing a corner plot with a different package (e.g. getdist) or you could upload the output samples file here and I'll have a quick look!
jovian-explorer
commented
2 years ago S5_wasp39b_ap6_bg7_Table_Save_ch00.txt -
# %ECSV 1.0
# ---
# datatype:
# - {name: time, datatype: float64}
# - {name: wavelength, datatype: float64}
# - {name: bin_width, datatype: float64}
# - {name: lcdata, datatype: float64}
# - {name: lcerr, datatype: float64}
# - {name: model, datatype: float64}
# - {name: residuals, datatype: float64}
# schema: astropy-2.0
time wavelength bin_width lcdata lcerr model residuals
57394.43603119213 1.575 0.07499999999999996 1.0155457685530875 3.924332707897192e-06 1.0144842484011813 0.0010615201519061923
57394.44155418567 1.575 0.07499999999999996 1.0109234952195083 3.927758523430964e-06 1.0144109574989513 -0.0034874622794429655
57394.447077179226 1.575 0.07499999999999996 1.013813117543826 3.940494313841179e-06 1.0143376665966246 -0.0005245490527985286
57394.45260017277 1.575 0.07499999999999996 1.0178055141826958 3.913363008511994e-06 1.0142643756943945 0.0035411384883012875
57394.45812316632 1.575 0.07499999999999996 1.013475484364688 3.902688545326441e-06 1.0141910847921645 -0.000715600427476426
57394.46364615987 1.575 0.07499999999999996 1.013333788089441 3.931072094011074e-06 1.0140661343468236 -0.000732346257382499
57394.46916915342 1.575 0.07499999999999996 1.0080727853784623 3.927104755250912e-06 1.0123825254885064 -0.004309740110044036
57394.47469214696 1.575 0.07499999999999996 1.010609193525835 3.920758159949212e-06 1.0087127900120798 0.001896403513755196
57394.48021514052 1.575 0.07499999999999996 1.0016713095392922 3.887685880760951e-06 1.0037338453289752 -0.002062535789683073
57394.48573813406 1.575 0.07499999999999996 0.9998934386437167 3.882424869632535e-06 0.9992762020026553 0.0006172366410613916
57394.491261127616 1.575 0.07499999999999996 0.9945161858137804 3.88537933706688e-06 0.995433177200871 -0.0009169913870905599
57394.49678412116 1.575 0.07499999999999996 0.9920916730374485 3.84648404891985e-06 0.9921266727075158 -3.499967006725946e-05
57394.50230711471 1.575 0.07499999999999996 0.9900026424453107 3.863176081272516e-06 0.9893527141314593 0.0006499283138513556
57394.50783010826 1.575 0.07499999999999996 0.9897443569720377 3.899580119760886e-06 0.987109617744246 0.0026347392277916715
57394.51335310181 1.575 0.07499999999999996 0.9884265969488489 3.867126093349606e-06 0.985396562451915 0.003030034496933953
57394.51887609535 1.575 0.07499999999999996 0.9836022610773048 3.872286654242764e-06 0.9842131115939431 -0.0006108505166383393
57394.52439908891 1.575 0.07499999999999996 0.9799830059474999 3.890840995311663e-06 0.9835590100968143 -0.003576004149314471
57394.52992208245 1.575 0.07499999999999996 0.9840476191631101 3.889046104800665e-06 0.9834340906025163 0.0006135285605938234
57394.535445076 1.575 0.07499999999999996 0.9830091064161582 3.882635900138912e-06 0.9838382341126964 -0.0008291276965382011
57394.54096806955 1.575 0.07499999999999996 0.9873624204231571 3.878897649632749e-06 0.9847713666350001 0.0025910537881570495
57394.5464910631 1.575 0.07499999999999996 0.9833714675736421 3.891069480234894e-06 0.9862334903382116 -0.002862022764569483
57394.552014056644 1.575 0.07499999999999996 0.9874583540139488 3.894361591560542e-06 0.9882247635113358 -0.0007664094973870439
57394.5575370502 1.575 0.07499999999999996 0.9890077170850284 3.879610829328565e-06 0.9907456732531236 -0.0017379561680951339
57394.56306004374 1.575 0.07499999999999996 0.9959971851199259 3.909305125299672e-06 0.9937974312690409 0.002199753850885
57394.5685830373 1.575 0.07499999999999996 0.9974242261025538 3.889034884168815e-06 0.9973830810058356 4.1145096718153695e-05
57394.57410603084 1.575 0.07499999999999996 0.9987364623954332 3.8940046603574495e-06 1.0015129822708855 -0.0027765198754522435
57394.57962902439 1.575 0.07499999999999996 1.0088620761025175 3.942941495606845e-06 1.0063986345540181 0.002463441548499423
57394.58515201794 1.575 0.07499999999999996 1.01141642519985 3.927430686819446e-06 1.010266143795426 0.0011502814044241916
57394.59067501149 1.575 0.07499999999999996 1.0179291243414301 3.906874700280908e-06 1.0122241109143841 0.005705013427045991
57394.596198005034 1.575 0.07499999999999996 1.0141294027352998 3.919288384899297e-06 1.0123588122355434 0.0017705904997564748
57394.60172099859 1.575 0.07499999999999996 1.0119162726279152 3.930436973215293e-06 1.0122855213332167 -0.0003692487053015281
57394.60724399213 1.575 0.07499999999999996 1.0058215234172458 3.952676079431498e-06 1.0122122304309866 -0.006390707013740782S5_dynesty_fitparams_ch00.csv -
# rp,per,t0,inc,a,u1,u2,c0,c1,scatter_mult
1.379843943777660231e-01,4.055253872377545754e+00,5.196662246646751582e-01,8.781071196366582399e+01,1.060021701306348341e+01,8.416056027000137219e-01,9.894983986880276916e-01,1.013348239416083985e+00,-1.327014084285402240e-02,1.820953615160138650e+00
-6.833866805777688569e-12,-7.993605777301127091e-15,-1.404404370575207395e-10,-9.362238984067516867e-09,-6.277982578239971190e-10,-2.111181229835779050e-10,-8.994627265224153234e-11,-4.862554803253260616e-12,-4.301120570815442079e-11,-0.000000000000000000e+00
1.140815220068702729e-11,1.154631945610162802e-14,1.026935203540801922e-10,3.444142748776357621e-09,6.052918166687959456e-10,3.809375037633344618e-10,7.709088922780438224e-11,2.940536703022189613e-12,1.345271116726109994e-11,0.000000000000000000e+00Aha, as I thought the emcee plots were indeed helpful. If you look at the chain and the corner plot from the emcee fit, you can see that the period and t0 parameters end up bumping into the edges of your prior and behave strangely. The reason for this is the t0 prior setting in your EPF which is centered around 0.5 which would've been from a NIRCam simulation while it should be around 57394.52 for the NIRSpec simulations. We apparently didn't update the t0 value in the template EPF when we updated all the ECFs for our new quickstart manual. emcee is better able to handle such a bad prior, but dynesty fails under such a bad prior apparently. I'll leave this issue open and make a quick patch, but for your own sake you can try updating the t0 line to the following:
t0 57394.52 'free' 57394.5 57394.55 USame error after making the above change as well
Traceback (most recent call last):
File "/home/keshav/Data/JWST-Sim/NIRSpec/run_eureka.py", line 29, in <module>
s5_meta = s5.fitlc(eventlabel, ecf_path=ecf_path, s4_meta=s4_meta)
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/s5_fit.py", line 173, in fitlc
meta = fit_channel(meta,time,flux,channel,flux_err,eventlabel,sharedp,params,log,longparamlist,time_units,paramtitles,chanrng)
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/s5_fit.py", line 257, in fit_channel
lc_model.fit(model, meta, log, fitter='dynesty')
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/lightcurve.py", line 156, in fit
fit_model = self.fitter_func(self, model, meta, log, **kwargs)
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/fitters.py", line 644, in dynestyfitter
plots.plot_corner(samples, lc, meta, freenames, fitter='dynesty')
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/plots_s5.py", line 203, in plot_corner
fig = corner.corner(samples, fig=fig, quantiles=[0.16, 0.5, 0.84], max_n_ticks=3,
File "/home/keshav/miniconda3/lib/python3.9/site-packages/corner/corner.py", line 226, in corner
return corner_impl(
File "/home/keshav/miniconda3/lib/python3.9/site-packages/corner/core.py", line 121, in corner_impl
raise ValueError(
ValueError: It looks like the parameter(s) in column(s) 9 have no dynamic range. Please provide a `range` argument.Yeah, I tried making that one change myself and I'm still getting issues with getting dynesty to converge, so there is clearly more issues with the EPF file than just that one - I'll be working on this for the next hour or so and will write here if/when I figure out a solution.
taylorbell57
commented
2 years ago Aha, so the issue is that the errors are greatly underestimated during Stage 3 because of how this tiny dataset was made. So using scatter_mult~1 gives uncertainties that are far too small and causes the fitters to have serious issues (and especially dynesty). For this particular sample data set, we'll likely want to use scatter_ppm instead and have the prior be 450 +/- 50 ppm or something like that. I'm running a test now and will confirm that resolves the issue
taylorbell57
commented
2 years ago Alright, in the pull request linked just above I've added several tweaks that should take care of this issue for you (and also added instructions on how to speed up the dynesty fit which you'd asked about earlier). The changes aren't live in the main branch yet, but you can copy-paste the edits into your own local files if you want to try them out (or do git checkout issue326 to checkout that branch). Once that pull request is approved, this issue will close automatically
jovian-explorer
commented
2 years ago All stages of NIRSpec running perfectly for tiny dataset now. Although, for a large rateints file, it gets killed.
(base) keshav@dev:~/Data/JWST-Sim/NIRSpec$ python3 run_eureka.py
WARNING: Unable to find an output metadata file from Eureka!'s S2 in the folder:
"/home/keshav/Data/JWST-Sim/NIRSpec/Stage1/"
Assuming this S2 data was produced by the JWST pipeline instead.
Starting Stage 2 Reduction
Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage1/
Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage2/S2_2022-05-19_wasp39b_run4/
Copying S2 control file
Found 1 data file(s) ending in rateints.fits
2022-05-19 14:55:07,649 - stpipe.EurekaSpec2Pipeline - INFO - EurekaSpec2Pipeline instance created.
2022-05-19 14:55:07,650 - stpipe.EurekaSpec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-05-19 14:55:07,650 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-05-19 14:55:07,651 - stpipe.EurekaSpec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.
2022-05-19 14:55:07,651 - stpipe.EurekaSpec2Pipeline.msa_flagging - INFO - MSAFlagOpenStep instance created.
2022-05-19 14:55:07,652 - stpipe.EurekaSpec2Pipeline.extract_2d - INFO - Extract2dStep instance created.
2022-05-19 14:55:07,654 - stpipe.EurekaSpec2Pipeline.master_background - INFO - MasterBackgroundNrsSlitsStep instance created.
2022-05-19 14:55:07,655 - stpipe.EurekaSpec2Pipeline.master_background.flat_field - INFO - FlatFieldStep instance created.
2022-05-19 14:55:07,656 - stpipe.EurekaSpec2Pipeline.master_background.pathloss - INFO - PathLossStep instance created.
2022-05-19 14:55:07,657 - stpipe.EurekaSpec2Pipeline.master_background.barshadow - INFO - BarShadowStep instance created.
2022-05-19 14:55:07,657 - stpipe.EurekaSpec2Pipeline.master_background.photom - INFO - PhotomStep instance created.
2022-05-19 14:55:07,658 - stpipe.EurekaSpec2Pipeline.wavecorr - INFO - WavecorrStep instance created.
2022-05-19 14:55:07,658 - stpipe.EurekaSpec2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-05-19 14:55:07,659 - stpipe.EurekaSpec2Pipeline.srctype - INFO - SourceTypeStep instance created.
2022-05-19 14:55:07,659 - stpipe.EurekaSpec2Pipeline.straylight - INFO - StraylightStep instance created.
2022-05-19 14:55:07,660 - stpipe.EurekaSpec2Pipeline.fringe - INFO - FringeStep instance created.
2022-05-19 14:55:07,661 - stpipe.EurekaSpec2Pipeline.pathloss - INFO - PathLossStep instance created.
2022-05-19 14:55:07,661 - stpipe.EurekaSpec2Pipeline.barshadow - INFO - BarShadowStep instance created.
2022-05-19 14:55:07,662 - stpipe.EurekaSpec2Pipeline.photom - INFO - PhotomStep instance created.
2022-05-19 14:55:07,663 - stpipe.EurekaSpec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2022-05-19 14:55:07,664 - stpipe.EurekaSpec2Pipeline.cube_build - INFO - CubeBuildStep instance created.
2022-05-19 14:55:07,665 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Extract1dStep instance created.
Starting file 1 of 1
Editing (in place) the waverange in the input file
KilledSo I'm pretty sure I encountered that same issue back in September before I got my work laptop which has a tonne of RAM. If you use a tool like htop to monitor your RAM while trying to run Stage 2, you'll likely see that it hits 100% usage (or at least close to that) and then your code gets killed. The issue is with how the simulated file is formatted, not with Eureka (for Stage 2, it's almost all functions from the STScI's jwst package anyway). The simulations for NIRSpec were provided as one large FITS file, and that file gets copied a few times in RAM to do things like sigma clipping, and your RAM likely can't handle that much. Real JWST observations will be broken into multiple smaller pieces that are less RAM intensive like the MIRI and NIRCam simulations. If you have access to a machine with more RAM you could try using that, otherwise there's not much you can do at this point besides trying a different instrument (MIRI and NIRCam should not give you this issue).
jovian-explorer
commented
2 years ago Right now I have only 8GBs of RAM. What RAM would be optimal for this? I am working on exoplanetary atmospheres and need the inferences from this pipeline to be matching the current models like PETITRADTrans, among others, as well as some observed data.
taylorbell57
commented
2 years ago I think I was encountering that same issue with 12GB of RAM, but I think it was fine when I upgraded to 32 GB (my work laptop now has 64GB which is quite likely overkill). Again, you won't need all that RAM for the real observations, just this one simulated file which is massive. Something in the ballpark of 8-12 GB will probably work plenty fine for the MIRI and NIRCam simulations which are more representative of what the real NIRSpec data will be like (in terms of RAM usage)
jovian-explorer
commented
2 years ago I have to work on the NIRSpec data for my project. I guess I should get some RAM then. Thanks for the help so far. I promise to trouble you again if any issues arise.
taylorbell57
commented
2 years ago Reopening this issue until Kevin approves my PR which will automatically close this issue
kevin218
commented
2 years ago Has anyone placed the Stage 2 outputs on Box? That might be the easiest solution.
On Thu, May 19, 2022, 1:00 PM Taylor James Bell @.***> wrote:
Reopened #326 https://github.com/kevin218/Eureka/issues/326.
— Reply to this email directly, view it on GitHub https://github.com/kevin218/Eureka/issues/326#event-6644339762, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFC2C7M6Z3HN4ROWQXE76D3VKZXTPANCNFSM5WG7RK6Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>
taylorbell57
commented
2 years ago I think they may be there, but the photom step will have been run in Stage 2 though, so the data quality may not be good
jovian-explorer
commented
2 years ago I have been trying to install/ run petitRADTRANS. Are there any other good alternatives, to help me interpret results that I get from Eureka?
taylorbell57
commented
2 years ago Sorry, that's not my expertise and it is beyond the scope of what we will be trying to do within Eureka - I know there are several good packages out there though (PLATON might be one?)
Instrument
NIRSpec (Stage 5)
What happened?
Well, the thing is that S5 runs okay for NIRSpec if I use lsq, or emce. But, for some reason, the emcee works for S5, while the dynesty doesn't. I mean, if I use emcee in S5, then the S6 runs perfectly well as well, but for Dynesty in S5, S5 itself fails. I didn't make any changes to the S5 ecf or epf except the fitting method, but somehow it doesn't run. This is for the tiny dataset. I haven't used the larger one again for now.
This is the ecf file for S5 -
This is the epf -
Error traceback output
What operating system are you using?
linux ubuntu 20.04
What version of Python are you running?
Python 3.9.7
What Python packages do you have installed?
Code of Conduct