search

LSSTDESC / ImageProcessingPipelines

Alert Production and Data Release image processing pipelines using the LSST Stack
BSD 3-Clause "New" or "Revised" License
3 stars 2 forks source link

U/heather/issue 134 #135

Closed heather999 closed 4 years ago

heather999 commented 4 years ago

This PR is meant to open a conversation about how we should update our ingestRefCatalog configuration for v19.0.0 as mentioned in #134 . I made a few updates based on the conversation with John P on Slack: https://lsstc.slack.com/archives/C2JPMCF5X/p1576522543077900?thread_ts=1576522428.077000&cid=C2JPMCF5X

There are some outstanding questions which require discussion. John was asking why we define ra_name to ra_smeared but then add ra to our list of extra_col_names. I'm not sure if this is associated with our update reference catalog back in April 2019. We seem to have both ra and ra_smeared in our reference catalog, so that must be reflected somewhere.

heather999 commented 4 years ago

After talking to Dominique and Jim, we've decided to keep the ingestRefCat for Run2.2i the same as Run2.2i. We don't apply parallax and proper motion in the sims, and do not want the stack to use those values (which are set to zero in our ref cat) anyhow. I've reverted the changes, so that the ingestRefCat config for Run2.2i is identical to what we used for Run2.1i. I'll do a fresh ingest of the reference catalog, run some checks and call this done.

heather999 commented 4 years ago

Completed a fresh ingest of the reference catalog using the same run2.1i ingestReferenceCatalog config, but renamed for run2.2 to leave a breadcrumb trail. I verified that the output FITS file names are identical in both the original Run2.1i ingest using w_2019_19 and the new Run2.2i ingest using v19.0.0. Using Jim's sims_ci_pipe code I checked both ingested reference catalogs at a variety of Ra Decs in the DC2 region and verified that the resulting reference cat objects are the same. A new tarball of the ingested reference catalog is now available at NERSC: /global/projecta/projectdirs/lsst/production/DC2_ImSim/Run2.2i/ref_cats/ref_cat-v1/dc2_run2.2i_reference_catalog_190513-v1.tar.gz

Here is one example from the notebook used to check on the reference catalog objects: 

run22_butler = dp.Butler("/global/cscratch1/sd/desc/DC2/Run2.2i/ingest_ref_cat/v19.0.0")

run22_ref_cats = RefCat(run22_butler)
centerCoord = lsst_geom.SpherePoint(61.855*lsst_geom.degrees, -35.79*lsst_geom.degrees)
print(centerCoord)
(61.855000, -35.790000)
run22_ref_cats(centerCoord, 'u')
<class 'lsst.afw.table.SimpleCatalog'>
      id           coord_ra           coord_dec      ... parallax radialVelocity
                     rad                 rad         ...                        
------------- ------------------ ------------------- ... -------- --------------
    855784450 1.0947293359291568 -0.6265160283473187 ...      0.0            0.0
  17254039554 1.1006122322343381 -0.6127632471198622 ...      0.0            0.0
  17265144834 1.1023647960785403 -0.6146372367612019 ...      0.0            0.0
  17280348162 1.1054333154510685 -0.6172418615096256 ...      0.0            0.0
  17282324482 1.1018568406192144 -0.6177768240541704 ...      0.0            0.0
  17296954370 1.0996995594049537 -0.6205442693415462 ...      0.0            0.0
  17309769730  1.096073539709165 -0.6230588797391078 ...      0.0            0.0
  17317073922  1.097851824990662 -0.6243081131592354 ...      0.0            0.0
  17317481474 1.0936270508474855 -0.6245565637749871 ...      0.0            0.0
  17320253442 1.1021163465501287 -0.6246993073150786 ...      0.0            0.0
          ...                ...                 ... ...      ...            ...
7176050719745 1.1057081013573569 -0.5999684282670598 ...      0.0            0.0
7176363948033 1.1032497302871485 -0.6052775181194902 ...      0.0            0.0
7176616150017 1.1034874084689574 -0.6030124255838285 ...      0.0            0.0
7180864139265 1.1059653447324407 -0.6031170552921761 ...      0.0            0.0
7181367144449 1.1008562084192997  -0.609146224116402 ...      0.0            0.0
7181402700801 1.1049041558487793 -0.5988665523795769 ...      0.0            0.0
7181980566529 1.1019100806890054 -0.6074874270949163 ...      0.0            0.0
7183232837633  1.103932022524591 -0.6074052430468064 ...      0.0            0.0
7183234684929 1.1027963404612178  -0.608485723567817 ...      0.0            0.0
7183255220225 1.1043509139142182  -0.605517155154132 ...      0.0            0.0
Length = 350549 rows
run21_butler = dp.Butler("/global/cscratch1/sd/desc/DC2/Run2.2i/ingest_ref_cat/w_2019_19")
run21_ref_cats = RefCat(run21_butler)
run21_ref_cats(centerCoord, "u")
<class 'lsst.afw.table.SimpleCatalog'>
      id           coord_ra           coord_dec      ... parallax radialVelocity
                     rad                 rad         ...                        
------------- ------------------ ------------------- ... -------- --------------
    855784450 1.0947293359291568 -0.6265160283473187 ...      0.0            0.0
  17254039554 1.1006122322343381 -0.6127632471198622 ...      0.0            0.0
  17265144834 1.1023647960785403 -0.6146372367612019 ...      0.0            0.0
  17280348162 1.1054333154510685 -0.6172418615096256 ...      0.0            0.0
  17282324482 1.1018568406192144 -0.6177768240541704 ...      0.0            0.0
  17296954370 1.0996995594049537 -0.6205442693415462 ...      0.0            0.0
  17309769730  1.096073539709165 -0.6230588797391078 ...      0.0            0.0
  17317073922  1.097851824990662 -0.6243081131592354 ...      0.0            0.0
  17317481474 1.0936270508474855 -0.6245565637749871 ...      0.0            0.0
  17320253442 1.1021163465501287 -0.6246993073150786 ...      0.0            0.0
          ...                ...                 ... ...      ...            ...
7176050719745 1.1057081013573569 -0.5999684282670598 ...      0.0            0.0
7176363948033 1.1032497302871485 -0.6052775181194902 ...      0.0            0.0
7176616150017 1.1034874084689574 -0.6030124255838285 ...      0.0            0.0
7180864139265 1.1059653447324407 -0.6031170552921761 ...      0.0            0.0
7181367144449 1.1008562084192997  -0.609146224116402 ...      0.0            0.0
7181402700801 1.1049041558487793 -0.5988665523795769 ...      0.0            0.0
7181980566529 1.1019100806890054 -0.6074874270949163 ...      0.0            0.0
7183232837633  1.103932022524591 -0.6074052430468064 ...      0.0            0.0
7183234684929 1.1027963404612178  -0.608485723567817 ...      0.0            0.0
7183255220225 1.1043509139142182  -0.605517155154132 ...      0.0            0.0
Length = 350549 rows