Closed geordie666 closed 2 years ago
@ashleyjross: I've created a full set of example files in:
/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/
I've verified that many of the desitarget.io
routines work with this data model for the randoms. For example (after reading some tiles of interest into tiles
):
hpdirname = "/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/"
io.read_targets_in_tiles(hpdirname, tiles=tiles)
but it would be useful if you can check everything works as expected for LSS catalog creation.
This looks perfect. We'll need to re-write some stuff to take advantage of it, but it should all be pretty trivial and speed everything up a lot in the end. Thanks!
-Ashley
From: Adam Myers @.> Date: Saturday, December 18, 2021 at 11:35 AM To: desihub/desitarget @.> Cc: ashleyjross @.>, Mention @.> Subject: Re: [desihub/desitarget] Functionality to split random catalogs by HEALPixel (PR #783)
@ashleyjrosshttps://github.com/ashleyjross: I've created a full set of example files in:
/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/
I've verified that many of the desitarget.io routines work with this data model for the randoms. For example (after reading some tiles of interest into tiles):
hpdirname = "/global/cscratch1/sd/adamyers/forashley/dr9/2.3.0.dev5334/randoms/resolve/"
io.read_targets_in_tiles(hpdirname, tiles=tiles)
but it would be useful if you can check everything works as expected for LSS catalog creation.
— Reply to this email directly, view it on GitHubhttps://github.com/desihub/desitarget/pull/783#issuecomment-997227441, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADLSM5UO6NAJX66VJZFKQFDURSZ4DANCNFSM5KKZ2BOA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
This PR adds functions and a command-line script to split random catalogs by HEALPixel. The resulting directory of HEALPixel-split random catalogs can be read using routines in
desitarget.io
such asread_targets_in_tiles()
.Splitting takes about 2.5 hours on an interactive node for the typical random catalog size of ~930 million randoms.