Add a way to shift offsets according to a reference CalDB

javierggt commented 1 year ago

Description

This PR makes it so the offsets can be adjusted to match the expectation after reprocessing using the latest alignment files in the CalDB.

The current master just plots the offsets resulting from the pipeline, but there is a time range that was processed with an earlier CalDB, and it would have smaller offsets if the observations were reprocessed. Since there will be no reprocessing soon, we need a way to display the offsets as if they had been reprocessed with the latest calalign files. This situation will happen whenever we update the calalign files.

This PR introduces the following changes:

the x_ray_src table has a new caldb_version field to keep track of the CalDB version used to compute the offsets. Whether this should be in the obs table is debatable. Accordingly, the observation.Observation class includes methods to determine the CalDB version and it also stores the calalign information for future use.
there are two new util functions
- utils.calalign_from_files which reads calalign files and produces a table of calalign matrices and offsets. This could also be in mica.
- utils.get_calalign_offsets uses calalign_from_files and the caldb_version in the cross-matches to shift the offsets so they match what would result from reprocessing using a reference CalDB (the latest CALALIGN files by default). This is only for display purposes, offsets on file are not changed.
Added a section to celmon web pages with plots of offsets with the most recent calalign files as reference.
Added a --use-latest-calalign option to calc_astrometry_rms.py (used for quarterly inputs).
In celmon plots, obsids with times after the reference calibration are shown with red markers.
While I was at it, fixed a few things in the celmon pages.

Interface impacts

Adds one field to a table in astromon file. Requires a new version of the file. For testing purposes I am using the file at /proj/sot/ska/jgonzalez/git/astromon/astromon-caldbver.h5

PROMOTION ACTIONS when this PR is promoted to production, the file needs to be promoted as well.

Testing

Unit tests

[x] Mac

Independent check of unit tests by Jean

[X] Linux

Functional tests

In order to produce the new astromon.h5 file, I first had to get the caldb version for all observations, which I did using this branch and this gist

For the following tests, I use a new astromon file:

export PYTHONPATH=/proj/sot/ska/jgonzalez/git/astromon
export ASTROMON_FILE=/proj/sot/ska/jgonzalez/git/astromon/astromon-caldbver.h5

Celmon

I ran the celmon script:

python -m  astromon.web.celmon --out /proj/sot/ska/www/ASPECT/jgonzalez/celmon/new --db-file $ASTROMON_FILE

and this is the output:

cal
mta

Documentation

Here is the updated documentation

calc_astrometry_rms

python -m  astromon.scripts.calc_astrometry_rms --use-latest-calalign

output:

Using all 'astromon_21' data
N srcs: 1070
RMS radius 0.464738
90 percentile radius = 0.62 arcsec
90 percentile radius for ACIS-S is 0.62 arcsec
90 percentile radius for ACIS-I is 0.78 arcsec
90 percentile radius for HRC-S is 0.60 arcsec
90 percentile radius for HRC-I is 0.60 arcsec
2.9 percent outside a 1 arcsec radius
---
2017:293:14:26:34.884 to 2022:293:20:26:34.884
N srcs: 319
RMS radius 0.51798016
90 percentile radius = 0.78 arcsec
90 percentile radius for ACIS-S is 0.78 arcsec
90 percentile radius for ACIS-I is 0.86 arcsec
90 percentile radius for HRC-S is 0.65 arcsec
90 percentile radius for HRC-I is 0.68 arcsec
2.8 percent outside a 1 arcsec radius
---
2018:293:20:26:34.884 to 2020:294:08:26:34.884
N srcs: 103
RMS radius 0.44146937
90 percentile radius = 0.65 arcsec
90 percentile radius for ACIS-S is 0.60 arcsec
90 percentile radius for ACIS-I is 0.76 arcsec
90 percentile radius for HRC-S is 0.61 arcsec
90 percentile radius for HRC-I is 0.62 arcsec
1.0 percent outside a 1 arcsec radius
---
2020:294:08:26:34.884 to 2022:293:20:26:34.884
N srcs: 162
RMS radius 0.60424274
90 percentile radius = 0.86 arcsec
90 percentile radius for ACIS-S is 0.85 arcsec
90 percentile radius for ACIS-I is 0.89 arcsec
90 percentile radius for HRC-S is 0.64 arcsec
90 percentile radius for HRC-I is 0.73 arcsec
4.3 percent outside a 1 arcsec radius

jeanconn commented 1 year ago

This makes sense to me, though where in this code do you pull out which matrices to use from a reference CALDB? I didn't immediately see that and you aren't getting quite a full line-by-line code review from me on this. Thanks!

javierggt commented 1 year ago

This makes sense to me, though where in this code do you pull out which matrices to use from a reference CALDB?

This is a good question. To "correct" from one reference CalDB to another, I actually subtract, because the effects of curvature will be small (these are a few arcseconds, after all). If I wanted to use the actual matrices to calculate the shift, I would use the matrices that I put in the dictionary.

If the question is where do I determine which CalDB version corresponds to a given time, that would be here (the last CalDB after removing CalDB versions after the one required, and sorting by version and start date)

sot / astromon