Using CatSim output to link to MAF "lightcurves"

[AshishMahabal]

Here is a recipe that can work with "real" lightcurves that can be extracted from CatSim, and combine it with the lightcurve capabilities metric that Steffano and Fed have been building.

Extract 5 (say) lightcurves for each type in CatSim, include it with the light-curve metric to get output light-curves for a given OpSim, at any (or every) point on the sky. Many other external quantitative checks can then be run on it.

[yoachim]

I'll make the comment that what we'd really like is to simply call all those "external quantitative checks" from within MAF. We'd like to avoid the step of dumping data out and computing important metrics outside the framework if we can.

I think there's a pretty high demand to be able to visualize the lightcurves that will be generated, I'll put that on my list of things we'd like to have soon in MAF.

[AshishMahabal]

Thats great, @yoachim. Scott has said that he can put together a notebook that shows how CatSim can be used in the next few days.

[yoachim]

Here's an example of using MAF to look at what light curves observed by LSST could look like: https://github.com/yoachim/ObservingStrategy/blob/master/notebooks/PullLightCurves.ipynb For this, I just assumed a saw-tooth light curve kinda like a z=0.5 SN Ia goes off continuously at some point in the sky. The resulting lightcurves that have more than 6 points above and SNR of 5 get plotted.

[drphilmarshall]

@yoachim - This notebook was great, thanks for making it! I think given other work along these lines (eg #123 ) it would be good to pull together the documentation a bit to make it easy for people to see how to calculate their metrics. Your PullLightCurves notebook looks to me like it belongs in the sims_maf_contrib area, as part of the documentation for all that stuff. Maybe linked from the wiki there? If you agree, you could issue yourself :-)

[yoachim]

HA! it's already there: https://github.com/LSST-nonproject/sims_maf_contrib/blob/8b35b6c211ff281d2fff38229df09ea37519d4da/tutorials/PullLightCurves.ipynb

[jhRho]

This notebook is great. That is exactly I need to get SN light curves. Jeonghee

[jhRho]

I am still learning python so I would like to obtain the final output of the light curve in ascii file which I can use other astronomical software to work on. This will speed the work I am doing for SNe. Can you add this function for me?

[yoachim]

@jhrlsst, notebook updated to dump each light curve to it's own ascii file.

Note that these are simple analytic saw-tooth light curves, but they should be close enough to real SN to fool other codes.

[jhRho]

I am going to figure out how to change the input files of light curve at some point based on real SN curves. At this point, what I need is list of day of visit (and # of visits) of a given magnitude source (and filter) based on Opsim. So I know how many data points the LSST survey will produce.

[fedhere]

Hi Jeonghee,

stefano valenti and I already created a metric that accepts an ascii lightcurve! i will get you the ref

On Thu, Nov 12, 2015 at 7:36 PM, Jeonghee Rho notifications@github.com wrote:

I am going to figure out how to change the input files of light curve at some point based on real SN curves. At this point, what I need is list of day of visit (and # of visits) of a given magnitude source (and filter) based on Opsim. So I know how many data points the LSST survey will produce.

— Reply to this email directly or view it on GitHub https://github.com/LSSTScienceCollaborations/ObservingStrategy/issues/127#issuecomment-156281110 .

---

dr. federica bianco James Arthur Fellow NYU-Center for Cosmology & Particle Physics NYU-Center for Urban Science & Progress New York, NY

http://cosmo.nyu.edu/~fb55/

---

//\ //||\ Please consider the environment before printing this e-mail

email checking schedule: 9-5 on the hour. otherwise sporadically.

[jhRho]

Great! thank you. Looking forward to it. It will be very useful. I am trying to run PullLightCurves.ipynb and my python doesn't recognize lsst.sims.maf.db.

In [6]: import lsst.sims.maf.db as db

ImportError Traceback (most recent call last)

in () ----> 1 import lsst.sims.maf.db as db I did install lsst.sims.maf with the command below. conda install lsst-sims-maf What did I miss? Thanks in advance. Jeonghee

[yoachim]

You still need to use EUPS to setup all the paths and environment variables before running lsst code. Looking at https://github.com/mjuric/conda-lsst it looks like you need to do a

source eups-setups.sh      # this activates EUPS (it's the equivalent of loadLSST.bash)
setup sims_maf    

[jhRho]

Good. I just solved the problem with help from Rahul. What you said was exactly right. Now I have an error at the stage "In [8]" In [37]: for fname in filters: ....: good = np.where(bundle.metricValues.data[0]['filter'] == fname) ....: plt.scatter(bundle.metricValues.data[0]['expMJD'][good]- dayZero, ....: bundle.metricValues.data[0]['fiveSigmaDepth'][good], ....: c = colors[fname], label=fname) ....: plt.xlabel('Day') ....: plt.ylabel('5$\sigma$ depth') ....: plt.legend(scatterpoints=1, loc="upper left", bbox_to_anchor=(1,1)) ....:
File "", line 6 plt.xlabel('Day') ^ IndentationError: unindent does not match any outer indentation level

I run ipython rather than as a notebook so I can see which line has a problem.

[fedhere]

the ascii transient metric. we did not make a notebook to go with it but i will

https://github.com/LSST-nonproject/sims_maf_contrib/blob/master/mafContrib/transientAsciiMetric.py

On Thu, Nov 12, 2015 at 10:58 PM, Peter Yoachim notifications@github.com wrote:

You still need to use EUPS to setup all the paths and environment variables before running lsst code. Looking at https://github.com/mjuric/conda-lsst it looks like you need to do a

source eups-setups.sh # this activates EUPS (it's the equivalent of loadLSST.bash) setup sims_maf

— Reply to this email directly or view it on GitHub https://github.com/LSSTScienceCollaborations/ObservingStrategy/issues/127#issuecomment-156316325 .

/|/\ |* Please consider the environment before printing this e-mail

[jhRho]

Thank for the program. Here are two stages of light curves in ascii I am talking about, and I want to understand the process. 1) I provide a supernova light curve as an input (in ascii). Then it seems that your program reads the ascii light curve. After that, should it be run with Catsim (or OpSim)? 2) then the final output light curve (output of PullLightCurves.ipynb), I would like to get the output light curve in ascii. I will run your program first which may give me ideas and understanding. Thanks again. I will let you know after I run the program.

[yoachim]

Looks like you have an indentation error (python cares about whitespace). Could be just a copy/paste issue? If you run the notebook, it will still stop and give full traceback error messages where it fails, you don't have to run outside the notebook.

[fedhere]

i will see if i can join the metric that peter made to get the ascii output to the metric that tekes the ascii input. i wont get to that before the end of the weekend though

On Thu, Nov 12, 2015 at 11:24 PM, Peter Yoachim notifications@github.com wrote:

Looks like you have an indentation error (python cares about whitespace). Could be just a copy/paste issue? If you run the notebook, it will still stop and give full traceback error messages where it fails, you don't have to run outside the notebook.

— Reply to this email directly or view it on GitHub https://github.com/LSSTScienceCollaborations/ObservingStrategy/issues/127#issuecomment-156318467 .

/|/\ |* Please consider the environment before printing this e-mail

[AshishMahabal]

Fed, thank for the progress on this important task. This is going to be useful to many.

[jhRho]

Fed, I am uploading the ascii file of SN2006X, an example of supernova light curve, which may be helpful when you work on the metric. I am also attaching a pdf file of the light curve. Currently squares indicates approximate of default LSST main survey (changing the filter every 3 days) and the data points from new strategy I want to test are marked in big red dots (more frequent visits for the same sky). I want to generate the same plots for more realistic LSST data points through opsim (catsim).

The ascii data are 'SN2006Xopt+nir_photo.dat',mjd, u,uerr,b,berr,g,gerr,r,rerr,i,ierr,y,yerr,j,jerr,h,herr,k,kerr.

[Uploading SN2006Xopt+nir_photo.dat…]() [Uploading SN2006Xopt+nir_photo.txt…]() [Uploading SN2006XObservingSSN.pdf…]()

Since pdf did not appear here, I added png file. Squares are my understanding of LSST data acquisition (1 filter for every 3 days) and red large dots are Cadence which may be optimized for SNe. Note no opsim data are used for this plot. Our goal is to make this plot from the output of Opsim/Catsim.

[jhRho]

I was able to successfully run PullLightCurves.ipynb. It is great. I am working on adding output statement of your light curve plots (basically to generate an ascii file of "day" and "mag"); I am still learning python. For ipython running, there was some issue that some statements have to be in one line etc (like for statement etc). Another problem I had was my computer hangs a few times with memory issue. I guess ipython notebook takes a lot of memory (it seems).

I need to find OpSim/CatSim ouput of main survey and deep survey (current default), respectively. Possibly after added the weather, dither etc. I got the data from http://ops2.tuc.noao.edu/runs/. Do you happen to know which runs they are or which ones are for main survey and deep survey? I don't see obvious README file. I plan to run PullLightCurves.ipynb using those data sets and see how they are different (at least errors should be different right?)

I will try the metric by Fed next.

Jeonghee PS. I am not sure if my attachments in previous comments were added correctly or not.

[rbiswas4]

I am not sure if you are using the sqlite database directly or the ASCII output you wanted.

sqlite: There is a table called summary that you will want to use. There is a column called propID and the values you would be interested in for WFD and DDF are 364 and 366.

If you are using the CSV we dumped this to: that csv only has the summary table. It should have the same propID column and you should select the same values to get main and deep observations.

Thanks, Rahul

On Fri, Nov 13, 2015 at 4:09 PM, Jeonghee Rho notifications@github.com wrote:

I was able to successfully run PullLightCurves.ipynb. It is great. I am working on adding output statement of your light curve plots (basically to generate an ascii file of "day" and "mag"); I am still learning python. For ipython running, there was some issue that some statements have to be in one line etc (like for statement etc). Another problem I had was my computer hangs a few times with memory issue. I guess ipython notebook takes a lot of memory (it seems).

I need to find OpSim/CatSim ouput of main survey and deep survey (current default), respectively. Possibly after added the weather, dither etc. I got the data from http://ops2.tuc.noao.edu/runs/. Do you happen to know what time runs they are or which ones are for main survey and deep survey? I don't see obvious README file. I plan to run PullLightCurves.ipynb using those data sets and see how they are different (at least errors should be different right?)

I will try the metric by Fed next.

Jeonghee PS. I am not sure if my attachment in previous comments added correctly or not.

— Reply to this email directly or view it on GitHub https://github.com/LSSTScienceCollaborations/ObservingStrategy/issues/127#issuecomment-156595207 .

[yoachim]

Hi @jhrlsst, If you're having memory issues with that notebook, you can try changing selecting a smaller subset of data from the database. right now sql='' is pulling the full 10 years, you can change that to something like sql='night < 1000' and that should ease up the memory used.

For that notebook, I pulled all the observations at ra,dec = (0,0). You can try changing those coordinates to other parts of the wide-fast-deep area (pretty much anything between a dec of -60 and 0, outside the galactic plane). Getting the positions of the deep drilling fields out of the sqlite files is a little convoluted, so here they are (in degrees): fieldRA|fieldDec 349.386444091797|-63.3210029602051 0.0|-45.5245056152344 53.0091438293457|-27.438943862915 34.3933982849121|-5.09032011032104 150.362350463867|2.83649897575378

[jhRho]

Thank you for information about deep field. I put night constraint as sql='night < 200' (ra = np.radians(3.24159) dec = np.radians(0.276115) Then I got the following results. 21 Observations total at this point (All SNR levels) 0 Observations in u 2 Observations in g 4 Observations in r 6 Observations in i 3 Observations in z 6 Observations in y 21 observations for 200 night observing run. This is only one LSST data point for 10 nights in average. It sounds very low (but it may be right number though). I recall to hear that LSST will cover entire visible sky for 3 nights. In addition, the filters are non-uniformly covered (zero obs in u and 6 obs in y) Then I got a plot of light curve showing day ranges from 0 to 10. What is definition of day here? The current definition seems to the number of observation instead of expMJD (relative to expMJDzero). Thus I expect "day" ranges 0 to 200.

[yoachim]

Note that np.radians is a function for converting degrees to radians. So those coordinates are right on the corner of the North Ecliptic Spur, which gets less coverage than the main survey. All the main survey will have negative dec values. Also, this field will be too close to the sun to be observed sometimes, so that's probably why there's such a large gap.

The plot auto-truncates the range, so I'm guessing there were no observations of that point between day ~9 and 200. Bump it up to 300 and you'll see lot's more points because the field becomes visible again.

And that plot is not a light curve, it's just showing the depth of all the observations. The x-axis is the MJD of the observation minus the MJD of the first observation.

[jhRho]

Thank you for clarifying the input position.

I don't quite understand the definition of x-axis (the MJD of the observation minus the MJD of the first observation should have unit of day, in fact that is what I want). If then, how can I convert to a light curve?

[yoachim]

Yes, the x-axis is in days, but the y-axis (on that first plot) is plotting the 5-sigma limiting depth of each observation, not the magnitude of an object.

OpSim doesn't include any SNe. So, if you want a light-curve you need to specify how it behaves with time yourself. In the notebook, I assumed something that looks like an SN Ia blows up every 45 (observer frame) days, and then I plotted up the light curves that were reasonably sampled.

My idea was that one could look at those curves and get an idea of what fraction are "well-observed", then we can use MAF to loop over the entire sky and make a map of the fraction of SN that we can expect to be well-observed as a function of RA and dec.

[jhRho]

I see you have commands later which creates light curves (day vs mag.). I will visit this issue later.

For now, I gave the position I need (now I see in numpy I can give the position in degree instead of radian). This is the position of SN2006X (RA: 12, 22, 55.05 and Dec: 15,49, 12.7) ra = np.radians(185.7293) dec = np.radians(15.8202) Querying database with constraint night < 300 bg.runAll() Found 230753 visits Running: [0] Completed metric generation. Running reduce methods. Running summary statistics. Completed. This seems to generate too many visits this time. 230753 visits which are not possible even for 10 year duration. Besides I gave constraint of "night <300". After that, I run "bundle.metricValues.data" command which gave the following error message. bundle.metricValues.data[0]['filter']

TypeError Traceback (most recent call last)

in () ----> 1 bundle.metricValues.data[0]['filter'] TypeError: 'NoneType' object has no attribute '**getitem**' When I go back to the position of (0,0), this error doesn't occur.

[jhRho]

Maybe the position I entered for SN2006X is within the deep fields. That may be the reason which gave many visits.

[yoachim]

That 230753 is all the visits before night 300, the slicer then does the work of figuring out which of those overlap the point you specified. It grabs everything because you can specify multiple points on the sky and change the field of view size.

LSST is in Chile. We only point above Dec=0 for select regions near the ecliptic and the galactic plane. The point you selected is not in the survey area. Check these plots for where we point: http://ops2.tuc.noao.edu:8888/multiColor?runId=94#C:%20NVisits

[lundmb]

The deep fields still shouldn't be that large. However, the number sounds pretty close to the total number of visits for the whole sky in a single year. Deep-drilling fields should be on the order of 10,000 visits over ten years, but the whole survey is on the order of a million visits.

[jhRho]

Ok. Thanks for pointing this out. I need to run visibility tool first (using CTIO tools etc). I thought Dec. of 23 degree should be visible, but I guess it depends on RA.

[jhRho]

Anyway it still shouldn't say "Found 230753 visits". It should say "0 visits" instead.

[yoachim]

It queried the summary table in the database with the sql constraint "where night < 300" and that returned 230753 visits. You can add things to the constraint like:

sql = 'night < 300 and fieldRA between 300 and 320 and fieldDec between 10 and 18'

Then it will find fewer visits.