search

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.
BSD 3-Clause "New" or "Revised" License
11 stars 7 forks source link

Update Run 1.2 static truth catalog (to include stars and adopt new data format) #328

Closed yymao closed 4 years ago

yymao commented 5 years ago

The current Run 1.2 static truth catalog (that has extinction) contains only galaxies. We need to add stars back to the static truth catalog.

Also, after some tests during hack sessions, it has become clear that the sqlite database (which is currently used) cannot support a large number of concurrent users. Hence, in this update, it may be a good idea to also update the underlying data format (e.g., to use just flat files).

This should be done for the variable truth catalog as well; however, the variable part can be directly taken care by https://github.com/LSSTDESC/gcr-catalogs/issues/269.

danielsf commented 5 years ago

Is there a base class of catalog reader that just reads in an HDF5 file on which I should base the reader for the HDF5 truth catalog reader?

(before I start writing from scratch and duplicating code)

yymao commented 5 years ago

Not really (I guess you have figured that).

katrinheitmann commented 5 years ago

@danielsf @yymao Hi both, has this happened? Is it still needed (since now we are moving rapidly to Run 2.1i)? If you could either update the status or close the issue that would be helpful. Thanks!

danielsf commented 5 years ago

This did not get done.

I do not have a sense whether or not it is still needed.

yymao commented 5 years ago

There were some people asking about this, but the requests haven't been (re-)appeared recently. I wonder if @rmandelb knows better.

wmwv commented 5 years ago

This should remain open.

katrinheitmann commented 4 years ago

@wmwv Hi Michael, Do you have an update on this? Another 6 months have passed. Thanks!

katrinheitmann commented 4 years ago

The half-year check-in: @wmwv @yymao Should this be still open? No action for a year right now. Thanks!

yymao commented 4 years ago

This is still not done for Run 1.2 unfortunately, but I think it is done for Run 2.2? @JoanneBogart would know about Run 2.2, but that's a separate question since this issue is about Run 1.2.

katrinheitmann commented 4 years ago

I guess my follow-up question is then: Will it be ever done and who would do it? Who is still working with the static output of Run 1.2? If we have more needs to Run 1.2 data products, we have to make a plan. I don't think this is on anybody's radar right now. (I could be wrong though!) Thanks!

jchiang87 commented 4 years ago

The star truth catalogs (that include MW extinction) are the same for Run1.2 as for Run2.2i. I believe those Run2.2i star truth catalogs are in the postgres database at NERSC.

katrinheitmann commented 4 years ago

Does the postgres implementation addresss Yao's worry about performance that he voiced at the start of the issue? Thanks!

JoanneBogart commented 4 years ago

The Postgres implementation (tables for star catalogs and sn) should handle concurrency much better than sqlite but it (concurrency performance) has not been tested. There are now also parquet files for star, sn and galaxy truth (also all for Run2.2i). A GCR reader is under development.

yymao commented 4 years ago

Truth star catalog is now also available in parquet/GCRCatalogs. Closing this.