Update Butler matching notebook for Run 1.2i

jiwoncpark commented 5 years ago

As discussed with @jchiang87, I modified the matching_stack.ipynb tutorial to reflect the changes that came with Run 1.2i. I also added some annotations at points where I got stuck changing the code to work for Run 1.2i.

Update summary:

Explains how to extract information about the repo from Butler, e.g. through getKeys() and queryMetadata()
Explains how to get available filter, tract, patch combinations
Updates the repo locations for src and deepcoadd in Jupyter-dev
Updates the src dataset key schema for Run 1.2i, e.g. raft --> raftName
Uses new column names for kronflux

I'm sorry if I missed something obvious, but I can't figure out why the number of matches between the drp catalog (from src) and protoDC2 is 0. I inferred from the original dataId in Jim's notebook that {'visit': 219976, 'raft': '2,2', 'sensor': '1,1'} for Run 1.1p were the same as {'visit': 219976, 'raftName': 'R22', 'detectorName': 'S11'} for Run 1.2i. Was I wrong?

fjaviersanchez commented 5 years ago

@jiwoncpark, I think that you can try to match directly to dc2_truth_run1.2_static so you don't have to worry about the rotation of the catalog. It is not finding any matches with the src_catalog because the regions don't overlap. You probably want to query a different sensor-raft (blue points are from the obj_catalog and orange are from the src_catalog):

fjaviersanchez commented 5 years ago

Another option is to query a different tract, patch (you can check which ones overlap with the visit in the notebook using: overlaps[(overlaps['visit']==219976) & (overlaps['detector']==94)]. E.g, using:

our_filter = 'r'
our_tract = 4639
our_patch = '6,6'

You get ~350 matches

jiwoncpark commented 5 years ago

Hi @fjaviersanchez, thank you for looking into this! Are you suggesting that I work with the truth catalog instead, which would entail turning it into a afw_table without the rotation and passing it into the matchRaDec function? If I were to stick with protoDC2, how would I go about figuring out which sensor, raft combo overlaps with a given region in protoDC2?

fjaviersanchez commented 5 years ago

You can use either protoDC2 or dc2_truth indistictively. The only difference would be that you don't have to rotate (you would still generate the afw.table/SourceCatalog object with dc2_truth as you said and in the function where you do the selection you avoid the rotation step).

The non-matching is a separate problem. In order to figure out which object catalog (coadd) overlaps with a given exposure (source catalog). In the overlaps table in your notebook you have all the information that you need (R22, S11 is detector=94).

This is because in this line: galaxy_catalog = region_selector(gc, band=our_filter, max_mag=mag_max)

You get rid of anything out of any galaxies lying outside of the patch that you queried (that's why you weren't finding any matches in the src catalog, because the piece of protoDC2 that you had didn't overlap with src).

rmandelb commented 5 years ago

@jiwoncpark - I just wanted to check in about this PR, since there hasn't been activity for a while. My impression from your exchange with @fjaviersanchez was that you may be planning additional changes before it's ready for review, so I was waiting on that - but please let me know if that's incorrect and I can seek out a reviewer now. Thanks!

reneehlozek commented 5 years ago

hi @jiwoncpark, just to check again, are there additional changes you are making on this PR? thanks!

jiwoncpark commented 5 years ago

Oh no, I missed Rachel's post from earlier. I'm so sorry! Yes, I recall I made some changes per suggestions from @fjaviersanchez but it seems I can't access the NERSC Jupyter server now (maintenance?) -- I'll request reviewers by this Thursday.

jiwoncpark commented 5 years ago

Thank you for your patience! In this version, ready for review, I've

implemented @fjaviersanchez 's suggestion of using the truth catalog instead and
added a method of finding the tract and patch combo for coadd that overlaps with a given src region, instead of an arbitrarily sky coordinate as I had in the previous version.

jiwoncpark commented 5 years ago

@wmwv has kindly agreed to review this PR!

jiwoncpark commented 5 years ago

I've also asked @jchiang87 (who wrote the original tutorial for Run 1.1) to review this!

reneehlozek commented 5 years ago

Great, good to hear, thanks!

reneehlozek commented 5 years ago

hi @jiwoncpark - is there an update on the changes requested by @wmwv for this PR?

jiwoncpark commented 4 years ago

I'll work on updating this for Run 2.xi.

reneehlozek commented 4 years ago

Perhaps we can hack these final updates based on 2.1/2.2i at the Spring Day @jiwoncpark?

jiwoncpark commented 4 years ago

Hi @reneehlozek, I'm sorry for the delay and that's a great idea -- I'll add this to the hack page. Thank you!

reneehlozek commented 4 years ago

hi @jiwoncpark, just wanted to check on the status of this notebook?

jiwoncpark commented 4 years ago

Hi @reneehlozek, I'm not sure if this tutorial is useful anymore, as I see that GCRCatalogs now publishes dc2_object_run2.1i_dr1*_matched and people can now just load that. I wrote the tutorial to teach myself how to match to truth, back when these weren't available. What do you think?

yymao commented 4 years ago

superseded by #135

LSSTDESC / DC2-analysis

Update Butler matching notebook for Run 1.2i #72