Document the simulation data files

[dkirkby]

This issue to is record useful into about the simulation data files that @esheldon has provided at nersc in /global/projecta/projectdirs/lsst/groups/WL/users/esheldon/nbrsim-outputs/collated/

[dkirkby]

Here is the info @esheldon has provided so far in the desc-blending slack channel:

• I dug up the catalogs for the "not-deblended" simulation, that showed some bias. I'll make these available. First I want to collate them into large catalogs to make it easier for people to use.
• One quirk of these sims is that I generated all images as separate entities, independent of others. They were all centered at the same RA,DEC
• They are not postage stamps, they are large images designed to be like DES coadds. Each catalog represents one of these.
• That's a 150G file, so I would recommend using the fitsio package to read subsets of columns, and in chunks.
• There is an id there file_id which tells you in which image the object was detected. Only do positional matching within an image.

[dkirkby]

Current contents of the directory:

-rw-r-----+ 1 esheldon lsst 149G Nov  9 10:43 run-003a-max001.fits
-rw-r-----+ 1 esheldon lsst 149G Nov  9 10:43 run-003b-max001.fits
-rw-r-----+ 1 esheldon lsst 149G Nov  9 10:43 run-003c-max001.fits
-rw-r-----+ 1 esheldon lsst 149G Nov  9 15:04 run-003d-max001.fits
-rw-r-----+ 1 esheldon lsst 149G Nov  9 15:06 run-003e-max001.fits
-rw-r--r--+ 1 esheldon des  580G Nov  9 23:52 run-003f-max001.fits

If you can't see these files, check that the output of groups includes lsst.

@esheldon What is the much larger file belonging to des?

[dkirkby]

The command fitsheader run-003a-max001.fits reports:

# HDU 0 in run-003a-max001.fits:
SIMPLE  =                    T / file does conform to FITS standard
BITPIX  =                   16 / number of bits per data pixel
NAXIS   =                    0 / number of data axes
EXTEND  =                    T / FITS dataset may contain extensions
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H

# HDU 1 in run-003a-max001.fits:
XTENSION= 'BINTABLE'           / binary table extension
BITPIX  =                    8 / 8-bit bytes
NAXIS   =                    2 / 2-dimensional binary table
NAXIS1  =                 3040 / width of table in bytes
NAXIS2  =             52433036 / number of rows in table
PCOUNT  =                    0 / size of special data area
GCOUNT  =                    1 / one data group (required keyword)
TFIELDS =                   80 / number of fields in each row
TTYPE1  = 'id      '           / label for field   1
TFORM1  = 'K       '           / data format of field: 8-byte INTEGER
TTYPE2  = 'number  '           / label for field   2
TFORM2  = 'J       '           / data format of field: 4-byte INTEGER
TTYPE3  = 'ra      '           / label for field   3
TFORM3  = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE4  = 'dec     '           / label for field   4
TFORM4  = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE5  = 'nimage_tot'         / label for field   5
TFORM5  = 'J       '           / data format of field: 4-byte INTEGER
TTYPE6  = 'fofid   '           / label for field   6
TFORM6  = 'K       '           / data format of field: 8-byte INTEGER
TTYPE7  = 'flags   '           / label for field   7
TFORM7  = 'J       '           / data format of field: 4-byte INTEGER
TTYPE8  = 'time_last_fit'      / label for field   8
TFORM8  = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE9  = 'box_size'           / label for field   9
TFORM9  = 'I       '           / data format of field: 2-byte INTEGER
TTYPE10 = 'obj_flags'          / label for field  10
TFORM10 = 'J       '           / data format of field: 4-byte INTEGER
TTYPE11 = 'psf_flags'          / label for field  11
TFORM11 = 'J       '           / data format of field: 4-byte INTEGER
TTYPE12 = 'psf_flux'           / label for field  12
TFORM12 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE13 = 'psf_flux_err'       / label for field  13
TFORM13 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE14 = 'psf_flux_s2n'       / label for field  14
TFORM14 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE15 = 'nimage_use'         / label for field  15
TFORM15 = 'J       '           / data format of field: 4-byte INTEGER
TTYPE16 = 'mask_frac'          / label for field  16
TFORM16 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE17 = 'psfrec_T'           / label for field  17
TFORM17 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE18 = 'psfrec_g'           / label for field  18
TFORM18 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE19 = 'gauss_flags'        / label for field  19
TFORM19 = 'J       '           / data format of field: 4-byte INTEGER
TTYPE20 = 'gauss_pars'         / label for field  20
TFORM20 = '6D      '           / data format of field: 8-byte DOUBLE
TTYPE21 = 'gauss_pars_cov'     / label for field  21
TFORM21 = '36D     '           / data format of field: 8-byte DOUBLE
TTYPE22 = 'gauss_g '           / label for field  22
TFORM22 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE23 = 'gauss_g_cov'        / label for field  23
TFORM23 = '4D      '           / data format of field: 8-byte DOUBLE
TTYPE24 = 'gauss_max_flags'    / label for field  24
TFORM24 = 'J       '           / data format of field: 4-byte INTEGER
TTYPE25 = 'gauss_max_pars'     / label for field  25
TFORM25 = '6D      '           / data format of field: 8-byte DOUBLE
TTYPE26 = 'gauss_max_pars_cov' / label for field  26
TFORM26 = '36D     '           / data format of field: 8-byte DOUBLE
TTYPE27 = 'gauss_s2n_w'        / label for field  27
TFORM27 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE28 = 'gauss_chi2per'      / label for field  28
TFORM28 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE29 = 'gauss_dof'          / label for field  29
TFORM29 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE30 = 'gauss_flags_r'      / label for field  30
TFORM30 = 'J       '           / data format of field: 4-byte INTEGER
TTYPE31 = 'gauss_s2n_r'        / label for field  31
TFORM31 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE32 = 'gauss_T_r'          / label for field  32
TFORM32 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE33 = 'gauss_psf_T_r'      / label for field  33
TFORM33 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE34 = 'mcal_flags'         / label for field  34
TFORM34 = 'K       '           / data format of field: 8-byte INTEGER
TTYPE35 = 'mcal_g  '           / label for field  35
TFORM35 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE36 = 'mcal_g_cov'         / label for field  36
TFORM36 = '4D      '           / data format of field: 8-byte DOUBLE
TTYPE37 = 'mcal_pars'          / label for field  37
TFORM37 = '6D      '           / data format of field: 8-byte DOUBLE
TTYPE38 = 'mcal_pars_cov'      / label for field  38
TFORM38 = '36D     '           / data format of field: 8-byte DOUBLE
TTYPE39 = 'mcal_gpsf'          / label for field  39
TFORM39 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE40 = 'mcal_Tpsf'          / label for field  40
TFORM40 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE41 = 'mcal_T  '           / label for field  41
TFORM41 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE42 = 'mcal_T_err'         / label for field  42
TFORM42 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE43 = 'mcal_T_r'           / label for field  43
TFORM43 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE44 = 'mcal_s2n_r'         / label for field  44
TFORM44 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE45 = 'mcal_g_1p'          / label for field  45
TFORM45 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE46 = 'mcal_g_cov_1p'      / label for field  46
TFORM46 = '4D      '           / data format of field: 8-byte DOUBLE
TTYPE47 = 'mcal_pars_1p'       / label for field  47
TFORM47 = '6D      '           / data format of field: 8-byte DOUBLE
TTYPE48 = 'mcal_pars_cov_1p'   / label for field  48
TFORM48 = '36D     '           / data format of field: 8-byte DOUBLE
TTYPE49 = 'mcal_T_1p'          / label for field  49
TFORM49 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE50 = 'mcal_T_err_1p'      / label for field  50
TFORM50 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE51 = 'mcal_T_r_1p'        / label for field  51
TFORM51 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE52 = 'mcal_s2n_r_1p'      / label for field  52
TFORM52 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE53 = 'mcal_g_1m'          / label for field  53
TFORM53 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE54 = 'mcal_g_cov_1m'      / label for field  54
TFORM54 = '4D      '           / data format of field: 8-byte DOUBLE
TTYPE55 = 'mcal_pars_1m'       / label for field  55
TFORM55 = '6D      '           / data format of field: 8-byte DOUBLE
TTYPE56 = 'mcal_pars_cov_1m'   / label for field  56
TFORM56 = '36D     '           / data format of field: 8-byte DOUBLE
TTYPE57 = 'mcal_T_1m'          / label for field  57
TFORM57 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE58 = 'mcal_T_err_1m'      / label for field  58
TFORM58 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE59 = 'mcal_T_r_1m'        / label for field  59
TFORM59 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE60 = 'mcal_s2n_r_1m'      / label for field  60
TFORM60 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE61 = 'mcal_g_2p'          / label for field  61
TFORM61 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE62 = 'mcal_g_cov_2p'      / label for field  62
TFORM62 = '4D      '           / data format of field: 8-byte DOUBLE
TTYPE63 = 'mcal_pars_2p'       / label for field  63
TFORM63 = '6D      '           / data format of field: 8-byte DOUBLE
TTYPE64 = 'mcal_pars_cov_2p'   / label for field  64
TFORM64 = '36D     '           / data format of field: 8-byte DOUBLE
TTYPE65 = 'mcal_T_2p'          / label for field  65
TFORM65 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE66 = 'mcal_T_err_2p'      / label for field  66
TFORM66 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE67 = 'mcal_T_r_2p'        / label for field  67
TFORM67 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE68 = 'mcal_s2n_r_2p'      / label for field  68
TFORM68 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE69 = 'mcal_g_2m'          / label for field  69
TFORM69 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE70 = 'mcal_g_cov_2m'      / label for field  70
TFORM70 = '4D      '           / data format of field: 8-byte DOUBLE
TTYPE71 = 'mcal_pars_2m'       / label for field  71
TFORM71 = '6D      '           / data format of field: 8-byte DOUBLE
TTYPE72 = 'mcal_pars_cov_2m'   / label for field  72
TFORM72 = '36D     '           / data format of field: 8-byte DOUBLE
TTYPE73 = 'mcal_T_2m'          / label for field  73
TFORM73 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE74 = 'mcal_T_err_2m'      / label for field  74
TFORM74 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE75 = 'mcal_T_r_2m'        / label for field  75
TFORM75 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE76 = 'mcal_s2n_r_2m'      / label for field  76
TFORM76 = 'D       '           / data format of field: 8-byte DOUBLE
TTYPE77 = 'sxflags '           / label for field  77
TFORM77 = 'J       '           / data format of field: 4-byte INTEGER
TTYPE78 = 'shear_true'         / label for field  78
TFORM78 = '2D      '           / data format of field: 8-byte DOUBLE
TTYPE79 = 'shear_index'        / label for field  79
TFORM79 = 'I       '           / data format of field: 2-byte INTEGER
TTYPE80 = 'file_id '           / label for field  80
TFORM80 = 'J       '           / data format of field: 4-byte INTEGER

[aimalz]

Sorry for jumping in so late! I took a look at the data to start preparing for the hack last week and remain somewhat lost. Where can I find information on the connection between these columns and the equations in the paper referenced in #2? Thanks.

[esheldon]

Those plots are from a different work, different type of catalogs. But the idea was to do a similar study with these sims and the metacalibration outputs

We want to do some tests such as looking at the mean shear as a function of distance to nearest neighbors; but as noted above this neighbor finding should only be done within a simulated field.

So here is the general idea:

import fitsio
import numpy

file_id = fitsio.read(filename, columns='file_id')
uids = numpy.unique(file_id)
for uid in uids:
    w, = numpy.where(file_id == uid)
    data = fitsio.read(filename, rows=w)

    # find neighbors and sort into radial bins
    # keep running sums of shapes `mcal_g` as well as`mcal_g_1p`, etc. for response calculation in each bin
    # calculate sum(g)/sum(R) in the bins.  We will also need to do selection effects but don't worry about that for now

[esheldon]

I will be on slack to help. I can also do some kind of audio/video chat, e.g. through slack.

[esheldon]

I changed group of that file to lsst

It should be no different than the others

[esheldon]

In terms of cuts, one should take

(flags ==0) & (mcal_T_r > mcal_Tpsf*0.5)

again, we will want to correct for selection effects, but ignore for now.

[fjaviersanchez]

@esheldon I assume that the mcal_T_r is the trace of the moments that result from metacal but what is the _r? What about the *_r_2m? the second moments?

[esheldon]

The _r means "round".

One can generate the round version of the model, and infer the s/n for that. This generally produces a smaller selection effect.