Compile HSC data

[chihway]

Compile HSC public data into TXPipe-usable format.

Javi and Judit have volunteered to do this.

[ellongley]

I have made an attempt at this in https://github.com/LSSTDESC/txpipe-reanalysis/tree/hsc_reanalysis_real, however, I wasn't able to reproduce all the cuts in the shear catalog paper. Additionally, we need to compile the n(z)'s.

[rmandelb]

@ellongley - I'm happy to consult on reproducing HSC cuts, if there are any questions you'd find useful to ask a DESC member who is also in HSC.

[fjaviersanchez]

Thanks @ellongley and @rmandelb! We can get the N(z)'s from @anicola and @damonge :). Also, what's the limitation for the cuts, are there columns missing? I thought we had all the cuts in this paper: https://ui.adsabs.harvard.edu/abs/2018PASJ...70S..25M/abstract (caveat: we made the cuts at the DB level but we didn't download some of the columns). I found a couple of issues myself at the time of trying to run this analysis in Fourier space, related to the way we create the masks (the depth maps with catalogs pre-selected in magnitude are incorrect and the bright object mask is missing) but I haven't found a way around this yet :(

[ellongley]

Thanks @fjaviersanchez and @rmandelb ! I pushed the cuts I've tried to a notebook here: https://github.com/LSSTDESC/txpipe-reanalysis/blob/hsc_reanalysis_real/hsc/notebooks/HSC%20Catalog%20Cuts.ipynb The cuts I couldn't apply because I couldn't find the columns are commented out and are:

['idetect_is_primary','ideblend_skipped',iflags_badcentroid','centroid_sdss_flags','iflags_pixel_edge','iflags_pixel_interpolated_center','iflags_pixel_saturated_center','iflags_pixel_cr_center','flags_pixel_bad','iflags_pixel_suspect_center','flags_pixel_clipped_any']

With the ones I have applied I have ~ 21 million objects ( so more than the 12.1 million that should be remaining for the shape catalog) Thank you again!

[rmandelb]

Hmmm. I could have sworn we created a flag that applies all the cuts from that paper, specifically so people wouldn't have to try to separately re-apply all the cuts themselves -- but I am struggling to find it in the database schema. (It's called weak_lensing_flag in the shape catalog paper, so I know it existed at some point!) I've put in a question to somebody who will definitely know.

In the meantime, I can say you definitely need to usewl_fulldepth_fullcolor to get consistent coverage with our full depth / full color cut (WL used a more strict one than was used for the PDR). That column does appear to be in your dataset, so you should apply it, unless it was already used to apply cuts when downloading from the database?

[rmandelb]

OK - Hironao Miyatake reminded me that if you want to reproduce the exact cuts, you don't need the WL flag because you can use an SQL query given on https://hsc-release.mtk.nao.ac.jp/doc/index.php/s16a-shape-catalog-pdr2/ to get the sample with all cuts applied via a database join.

Otherwise, you do indeed need to separately apply all cuts, unfortunately.

[ellongley]

Hi all, thank you for the help. I am coming back to this after getting a couple other things set up. I used the following query based on the s16 instructions (with additional columns, not including all here):

SELECT a.ishape_hsm_regauss_flags FROM s16a_wide.meas2 a inner JOIN s16a_wide.weaklensing_hsm_regauss b USING (object_id) inner JOIN s16a_wide.meas c USING (object_id) inner JOIN s16a_wide.photoz_ephor_ab d USING(object_id) WHERE s16a_wide.search_hectomap(c.skymap_id) ;

(for each field) this resulted in the following number of objects in each bin, using the photoz_ephor_ab "best" option: 11,942,756 total 2,986,305 0.3<z_best<0.6 3,018,002 0.6<z_best<0.9 2,270,349 0.9<z_best<1.2 1,314,138 1.2<z_best<1.5

which are more objects than in the shape catalog paper. In the query I pulled all columns to reproduce the cuts from table four and checked they were satisfied. For each object wl_fulldepth_fullcolor=True. Might there be an additional mask or cut that I need to apply? Thank you!

[HironaoMiyatake]

Are you referring to Hikage et al? I actually made the same query as yours for XMM and got the same number as my "master" shape catalog I made a while ago, so I think your catalog and mine is consistent. I then guessed there might be an additional cut in Hikage et al. applied to the "master" catalog, but I could not find. I might have to ask him if he added such a cut, but can you confirm if you are comparing your number to Hikage et al.?

[ellongley]

So I guess I'm technically working on the real-space so Hamana et al https://arxiv.org/pdf/1906.06041.pdf but it should be the same sample, so expecting to get the numbers in table 1 of Hikage et al.

[HironaoMiyatake]

I see. Then let me talk to both of them.

[ellongley]

Thank you @HironaoMiyatake, to summarize our conversation here, the numbers we have for this catalog are correct, and we should press forward with these cuts.

[ellongley]

This means that the differences in #16 are from something else.