search

LSSTDESC / SprintWeek2020

Meeting repository for the LSST DESC 2020 Sprint Week
Apache License 2.0
2 stars 0 forks source link

[SPRINT] Create parquet files for Skysim5000 #63

Closed yymao closed 3 years ago

yymao commented 3 years ago

Create parquet files for Skysim5000

The Skysim5000 is getting ready to be released via GCRCatalogs (cf. https://github.com/LSSTDESC/gcr-catalogs/pull/496). This sprint aims to generate a few parquet files for Skysim5000, with the hope of improving access performance.

Contacts: @yymao @JoanneBogart @evevkovacs Day/Time: TBD Main communication channel: #desc-dc2-data-access, #desc-co GitHub repo: N/A

Goals and deliverable

Parquet files for Skysim5000, with some performance tests.

Resources and skills needed

Basic experience with GCRCatalogs. Bonus points for Dask.

Detailed description

For more detail about Skysim5000, please see talk by @evevkovacs --- you can find the recording in this Slack thread.

yymao commented 3 years ago

Done! File paths can be found on https://confluence.slac.stanford.edu/x/1KbgEQ