Create PostgreSQL tables and parquet files for Run3.1i truth

[JoanneBogart]

Truth tables in the form of sqlite files have been created for agn, lensed agn, lensed sne, and lensed host galaxies. They should be ingested into PostgreSQL and also translated to parquet.

[JoanneBogart]

For PostgreSQL the steps involved for each table are

1. Extract schema information from sqlite; create comparable table in the PostgreSQL database. Datatype names differ. Also tables with ra,dec columns (truth summary, one for each object type) should have an additional column of type earth, typically called coord, which may be indexed and used for spatial searches.
2. Extract data to a csv file
3. Ingest data from csv into the PostgreSQL table
4. If there is a coord column, compute values from ra,dec and update the database table
5. Create indexes as needed

For parquet, use a python script previously used for other truth tables. It may need some tweaking.

Finally, validate: check that information has been preserved. For example, one might

• check that row counts are identical
• for a few rows, check that values are identical (or within tolerance for floating point)
• for truth summary tables, plot (ra, dec) for the three forms and compare
• pick some columns to be histogrammed for the three forms and compare

[JoanneBogart]

All Run3.1i truth tables are in PostgreSQL:

• Schema agn_truth: tables summary_truth, agn_auxiliary_info, agn_variability_truth
• Schema lensed_agn_truth: tables summary_truth, lensed_agn_variability_truth
• Schema lensed_host_truth: table summary_truth
• Schema lensed_sne_truth: tables summary_truth, lensed_sn_variability_truth

Counts look ok.

[JoanneBogart]

Parquet files have been created.