LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.
BSD 3-Clause "New" or "Revised" License
11 stars 7 forks source link

Keep object ID and store tract/patch info for metacal parquet files #385

Closed yymao closed 4 years ago

yymao commented 4 years ago

As title. While it takes a bit of space, this will save us much trouble debugging.

EiffL commented 4 years ago

ok well, that won't work, it will complain that the id is duplicated between the original catalog and the metacal catalog, that was the entire reason I did this .....

EiffL commented 4 years ago

But for checking, we can probably rename the id column to metacal_id and then we can keep it.

yymao commented 4 years ago

Who is the "it" that complains about the duplicated column?

EiffL commented 4 years ago

The composite catalog reader, when the two catalogs have columns with the same name

yymao commented 4 years ago

It won't / shouldn't complain -- we made the composite feature so that we can overwrite columns in the main catalog.

Can you show a case where it complains?

EiffL commented 4 years ago

Ah.... hummmm :-/.... if you say so ^^' Can we test your PR on one example and see what happens? I need to remind myself where to find data files...

yymao commented 4 years ago

I mean, the old metacal files have id column and it works fine in GCR/composite. See https://github.com/LSSTDESC/gcr-catalogs/blob/master/GCRCatalogs/dc2_metacal.py#L65

Removing the id column is a new thing (and I actually had to change the reader for this, see https://github.com/LSSTDESC/gcr-catalogs/pull/419/files#diff-a607bcf79b5184cd48390a8054da24e8L65, which BTW is pending a review from a good friend of mine :) )

yymao commented 4 years ago

@EiffL, would it be ok to merge this?

EiffL commented 4 years ago

yes, this worked last time I tried it

yymao commented 4 years ago

Thank you @EiffL! Merging..

johannct commented 4 years ago

if this needs to go to DR6, DC2-prod needs a patch release quick..... possibly cherry picking

yymao commented 4 years ago

It's not critical since it doesn't affect science but just helps with debugging.