search

desihub / desitarget

DESI Targeting
BSD 3-Clause "New" or "Revised" License
18 stars 23 forks source link

Functionality to split random catalogs by HEALPixel #783

Closed geordie666 closed 2 years ago

geordie666 commented 2 years ago

This PR adds functions and a command-line script to split random catalogs by HEALPixel. The resulting directory of HEALPixel-split random catalogs can be read using routines in desitarget.io such as read_targets_in_tiles().

Splitting takes about 2.5 hours on an interactive node for the typical random catalog size of ~930 million randoms.

geordie666 commented 2 years ago

@ashleyjross: I've created a full set of example files in:

/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/

I've verified that many of the desitarget.io routines work with this data model for the randoms. For example (after reading some tiles of interest into tiles):

hpdirname = "/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/"
io.read_targets_in_tiles(hpdirname, tiles=tiles)

but it would be useful if you can check everything works as expected for LSS catalog creation.

coveralls commented 2 years ago

Coverage Status

Coverage decreased (-0.2%) to 56.533% when pulling 605d478c0ff24106ff53f43939ff709d1428d2eb on ADM-pix-rands into 3cf1c4b723da315508ea8cd9d9be8240034b23dd on master.

ashleyjross commented 2 years ago

This looks perfect. We'll need to re-write some stuff to take advantage of it, but it should all be pretty trivial and speed everything up a lot in the end. Thanks!

-Ashley

From: Adam Myers @.> Date: Saturday, December 18, 2021 at 11:35 AM To: desihub/desitarget @.> Cc: ashleyjross @.>, Mention @.> Subject: Re: [desihub/desitarget] Functionality to split random catalogs by HEALPixel (PR #783)

@ashleyjrosshttps://github.com/ashleyjross: I've created a full set of example files in:

/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/

I've verified that many of the desitarget.io routines work with this data model for the randoms. For example (after reading some tiles of interest into tiles):

hpdirname = "/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/"

io.read_targets_in_tiles(hpdirname, tiles=tiles)

but it would be useful if you can check everything works as expected for LSS catalog creation.

— Reply to this email directly, view it on GitHubhttps://github.com/desihub/desitarget/pull/783#issuecomment-997227441, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADLSM5UO6NAJX66VJZFKQFDURSZ4DANCNFSM5KKZ2BOA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>