spacetelescope / romancal

Python library to process science observations from the Nancy Grace Roman Space Telescope
https://roman-pipeline.readthedocs.io/en/latest/
Other
31 stars 28 forks source link

pass exclude/ignore options to table meta comparison #1226

Closed braingram closed 4 months ago

braingram commented 5 months ago

This PR allows passing ignore paths that include items in astropy.Table metadata by parsing the exclude_paths handled by DeepDiff and passing them through to the diff called on the table metadata. A test is added for the new feature.

@schlafly would you try this with the table date metadata issue you mentioned? Thanks!

Checklist

schlafly commented 5 months ago

Yes. I am regenerating the regtest files right now, though, so it will be a bit. Here was the approach I took: https://github.com/schlafly/romancal/blob/ba1e79d2893602a9f52ca398fa4d8d5ed7efcf32/romancal/regtest/regtestdata.py#L617 Just making sure I follow, my version does the one dumb necessary thing, and your version takes the whole exclude_paths and copies them so they apply to the Table metadata as well. In particular, roman.meta.date is in there, and we hope that handles meta.date here as well?

braingram commented 5 months ago

Thanks! No rush from my end. I gave this a look (as a reprieve from hunting down memory usage) and saw what might be a relatively general approach.

For this PR let's say there is a astropy.Table for the source_catalog attribute of a datamodel. If that table contains a meta.date that you want compare_asdf to ignore you could modify ignore_asdf_paths to include:

source_catalog.meta.date

and it would ignore specifically the date metadata entry for the source_catalog table.

braingram commented 4 months ago

This can likely be "revived" if the changes are wanted. Since it currently has conflicts I'm closing it until that point.