Add doc for converting from sqlite to other formats and draft noteboo…

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.

BSD 3-Clause "New" or "Revised" License

11 stars 7 forks source link

Add doc for converting from sqlite to other formats and draft noteboo… #424

Closed JoanneBogart closed 3 years ago

JoanneBogart commented 3 years ago

fix #423

JoanneBogart commented 3 years ago

Still to be resolved:

should the module parquet_utils, currently in repo sims_TruthCatalog, be moved here? It creates a parquet file from SQLite and can do some rudimentary checks. The problem with leaving it in sims_TruthCatalog is that that repo is dependent on the stack, which otherwise would not be necessary to run the module-level script.
What belongs in the verification notebook? So far comparing counts and plotting coverage (for truth summary tables) are implemented. Other possibilities include comparing light curves (partially done) for a particular object, comparing histograms of column values, comparing values in rows.

@yymao If you have a chance to take a look I'd be interested in your opinions on the above. Thanks!

yymao commented 3 years ago

I don't have strong opinion regarding the two points you raised:

As long as the location of parquet_utils is well documented that's oK. If we keep it in sims_TruthCatalog, maybe you can add a note in this instruction saying one can run this script without stack.
Your proposal of contents in the verification notebook sounds reasonable. If there's any validation checks that can be done automatically (i.e., without the need of human inspection), those can be written as a script or even as part of the generation script (e.g., with a --validate option to turn it on).

I took a quick look of convert_sqlite.md. Most of it looks good.

However, I can't quite follow the "Creating the PostgreSQL schema" section though. For example, my_create.sql is not used in the following steps. I don't know what psql is and how to run it. Is desc_dc2_drp a table name? The section mentions "If original table has ra,dec columns, add a column of type earth"; how does one do so?

JoanneBogart commented 3 years ago

Thanks @yymao ! I wrote up the notes mostly for my own use so they do assume familiarity with various things, but it would be better if they were more self-contained. I will aim for that in the next round of updates.

JoanneBogart commented 3 years ago

I've updated the documentation to be less cryptic. The notebook includes a direct check for ra, dec values in the summary tables in addition to plots. Those checks and others like them should at some point be moved into a script.

JoanneBogart commented 3 years ago

@yymao There is no rush, but please have a look when you get the chance. I'm not planning to do any more work on this (other than fixing issues you find) in the near future and would like to merge what's here. Thanks!

JoanneBogart commented 3 years ago

Thanks, @yymao :-)