Epoch parameter is not being used for Gaia catalog so astrometry is wrong

[stscijgbot-hstdp]

Issue HLA-1124 was created on JIRA by Rick White:

The EPOCH parameter for the gsss webservice is used to specify the date of the observation. The service applies the known proper motions and parallax to update the Gaia catalog reference positions to the observation epoch. That allows the astrometric matching to determine an accurate updated WCS for the image.

In the latest test version, apparently the EPOCH parameter is not being used or is being incorrectly specified. As a result, the positions in the Gaia astrometric reference catalog are always for epoch 2016. A detailed summary that leads to this conclusion is given in the comments below.

This will lead to much poorer astrometry in the HAP products and is a high priority to fix.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

As a fairly simple example, I am looking closely at image hst_15446_59_acs_wfc_jdrz59. I have examined the results for this image in both the recent test samples using the gsss and gssstest version of the web service.

Here are links to the interactive display for the 2023-09-13_gsss version and the 2023-09-14_gssstest version.

The Gaia reference catalog is in file hst_15446_59_acs_wfc_f606w_jdrz59_gaia_sources.reg and has 7 Gaia sources:

#RA,DEC
54.9688391244112,-2.11345211450208
54.945329794617,-2.11513075668512
54.959508865931,-2.09533778075101
54.9723658519912,-2.11321617354845
54.953525407635,-2.13003083189077
54.97188054688,-2.11309904048647
54.9786027745196,-2.097886787564

The reference catalog reported for the two versions is identical (which is itself a clue that something is wrong, because the catalog should be slightly different with the inclusion of parallax in the newer service).

This list of Gaia EDR3 source positions was compared with the results from Brian McLean's simple test interface to the GSSS catalogs. These parameters are used to search the hst_15446_59_acs_wfc_jdrz59 sky area:

||Parameter||Value|| |RAmin|54.940| |RAmax|54.983| |DECmin|-2.135| |DECmax|-2.090| |Catalog|GaiaEDR3| |Epoch|2019.117314|

The epoch is derived from the hst_15446_59_acs_wfc_jdrz59 date of observation (2019-02-13 02:14:21 to 02:30:04). Here is a subset of the parameters that are returned:

source_id,ra,ra_error,dec,dec_error,parallax,pmra,pmdec
3250909056134553088,54.9688391244112,31.23712,-2.11345211450208,36.93051, , ,
3250908884336492928,54.9453329242454,4.500015,-2.11513423225992,4.559355,-1.82524204693432,3.05067060655313,-4.15193866159561
3250909266588504192,54.9595159107546,2.991025,-2.09533663054662,3.630707,1.2824246701404,8.52443095441108,1.42541141785927
3250908991711890688,54.9723658519912,32.13297,-2.11321617354845,32.10233, , ,
3250908575098745344,54.9535248152358,1.133269,-2.13005774420234,1.235946,3.28510938226273,0.326205349239861,-30.8306784661846
3250908991711890560,54.9718807466334,2.127836,-2.11309815405796,2.071113,0.458378140716482,0.371434367655107,1.05839361980687
3250909090494302720,54.9786108171998,0.3725007,-2.09788672547489,0.3807495,0.62586316489199,9.47418029497469,0.119097423464696

Those entries are sorted into the same order as the hst_15446_59_acs_wfc_f606w_jdrz59_gaia_sources.reg file. Below a table version that is easier to read is given.

A modified version of the gsss query was also run with exactly the same search parameters but with the Epoch parameter omitted. If no epoch is specified, the service returns the values from the catalog with no proper motion correction. Here are the results in that case:

source_id,ra,ra_error,dec,dec_error,parallax,pmra,pmdec
3250909056134553088,54.9688391244112,13.54903,-2.11345211450208,23.90985, , ,
3250908884336492928,54.945329794617,1.030988,-2.11513075668512,0.9787956,-1.82524204693432,3.05067060655313,-4.15193866159561
3250909266588504192,54.959508865931,0.6337197,-2.09533778075101,0.7175019,1.2824246701404,8.52443095441108,1.42541141785927
3250908991711890688,54.9723658519912,2.358504,-2.11321617354845,1.895914, , ,
3250908575098745344,54.953525407635,0.2734624,-2.13003083189077,0.2709915,3.28510938226273,0.326205349239861,-30.8306784661846
3250908991711890560,54.97188054688,0.5193594,-2.11309904048647,0.4641822,0.458378140716482,0.371434367655107,1.05839361980687
3250909090494302720,54.9786027745196,0.08970115,-2.097886787564,0.08283798,0.62586316489199,9.47418029497469,0.119097423464696

It is probably difficult to read, but the gsss query is identical to the positions in the HAP reference file when the Epoch parameter is omitted. That is incorrect.

Here is a tabular summary of the data which should be a little easier to read:

||Source||Value||HAP||gssstest with epoch||gssstest without epoch|| |1|RA | 54.945329794617 | 54.9453329242454 | 54.945329794617 | |2|RA | 54.953525407635 | 54.9535248152358 | 54.953525407635 | |3|RA | 54.959508865931 | 54.9595159107546 | 54.959508865931 | |4|RA | 54.9688391244112 | 54.9688391244112 | 54.9688391244112 | |5|RA | 54.97188054688 | 54.9718807466334 | 54.97188054688 | |6|RA | 54.9723658519912 | 54.9723658519912 | 54.9723658519912 | |7|RA | 54.9786027745196 | 54.9786108171998 | 54.9786027745196 | |1|Dec | -2.11513075668512 | -2.11513423225992 | -2.11513075668512 | |2|Dec | -2.13003083189077 | -2.13005774420234 | -2.13003083189077 | |3|Dec | -2.09533778075101 | -2.09533663054662 | -2.09533778075101 | |4|Dec | -2.11345211450208 | -2.11345211450208 | -2.11345211450208 | |5|Dec | -2.11309904048647 | -2.11309815405796 | -2.11309904048647 | |6|Dec | -2.11321617354845 | -2.11321617354845 | -2.11321617354845 | |7|Dec | -2.097886787564 | -2.09788672547489 | -2.097886787564 |

The gssstest with epoch column has the correct values for this observation, but the actual values in the HAP column agree exactly with the no epoch results.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

This might be an old problem. I remember looking to see how the gsss web service gets called, and noticing that if there is an error returned, it repeats the call with the epoch parameter omitted (see this code line in astrometric_utils.py module). Brian's service expects the epoch to be a decimal year (e.g., 2019.1), and it returns an error if it is called with a crazy value.  For example, if you pass it an MJD value of 55000, or a string value with the date like "2019-02-13" then it will return an error.

So my best guess is that either the  parameter has gotten omitted in the call to {}astrometric_utils.get_catalog(){}, or it is being passed but has the wrong units.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

I took a quick look and I want to document another issue.  The user_epoch parameter can be a string or a float.  This is not good practice.  Use functools.singledispatch or a similar coding technique.  Also, check the resultant epoch is a decimal year or it is an error.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

See discussion of 18 September 2023 in team_burgundy.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena I'm looking at the code and am pretty mystified by the problem. Apparently the user_epoch parameter never gets specified when create_astrometric_catalog is called, so it defaults to using the value of DATE-OBS in the primary header. And it converts that date to decimal years. So the code all looks correct.

I guess it would be helpful to run this with debugging turned on, which will trigger print statements in the get_catalog and get_catalog_from_footprint functions that will tell exactly what URL is used for the service request.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White I hope I have not been working at cross purposes with you.

As Rick has examined, I looked at visit hst_15446_59_acs_wfc_jdrz59 which is comprised of two images: jdrz59lxq_flc.fits and jdrz59m2q_flc.fits. Having said this I printed to stdout the input values before the catalog service was invoked.  Below you see the inputs from the program, as well as the output information.

Use of GSSS vs GSSSTEST via Programmatic Interface The input values to the catalog services were the same for all tests.

CRVAL :  54.96905704495079  -2.115762660869403 Radius:  0.041834218730393224 Catalog: GAIAeDR3 Epoch:   2019.1178082191782 (only 2019.118 used for service) base_spec: 'RA=54.96905704495079&DEC=-2.115762660869403&SR=0.041834218730393224&FORMAT=CSV&CAT=GAIAeDR3&MINDET=5' spec: 'RA=54.96905704495079&DEC=-2.115762660869403&SR=0.041834218730393224&FORMAT=CSV&CAT=GAIAeDR3&MINDET=5&EPOCH=2019.118'

use_footprint = False ==> indicates routine get_catalog() is used

For my tests done in debug mode, I checked the serviceUrl to make sure of the service in use (i.e., gsss vs gssstest).  For example,

Pdb) serviceUrl 'http://gssstest.stsci.edu/webservices/vo/CatalogSearch.aspx?RA=54.96905704495079&DEC=-2.115762660869403&SR=0.041834218730393224&FORMAT=CSV&CAT=GAIAeDR3&MINDET=5&EPOCH=2019.118'

 

TEST 1) When no EPOCH was specified, the coordinates for the fifteen output         entries were identical between the GSSS and GSSSTEST.

TEST 2) When the EPOCH was specified, the eleven entries with parallax and         proper motion values were different between GSSS and GSSSTEST.         Using the NOEPOCH as a base reference.

 

 

              NOEPOCH                               GSSS                              GSSSTEST
       ra               dec                 ra               dec                 ra               dec
    float64           float64            float64           float64            float64           float64
---------------- -----------------  ---------------- -----------------  ---------------- -----------------
G54.9688391244112 -2.11345211450208* 54.9688391244112 -2.11345211450208  54.9688391244112 -2.11345211450208
G54.97188054688   -2.11309904048647  54.9718808688024 -2.1130981238      54.9718807467042 -2.11309815385628
G54.9723658519912 -2.11321617354845* 54.9723658519912 -2.11321617354845  54.9723658519912 -2.11321617354845
G54.9786027745196 -2.097886787564    54.978610985716  -2.09788668441239  54.9786108190063 -2.0978867254522
G54.953525407635  -2.13003083189077  54.9535256903605 -2.13005753468395  54.953524815298  -2.1300577500773
G54.959508865931  -2.09533778075101  54.9595162539752 -2.09533654618634  54.9595159123801 -2.095336630275
G54.945329794617  -2.11513075668512  54.9453324386381 -2.11513435272533  54.9453329248271 -2.1151342330511
54.961920626249  -2.08641622574526* 54.961920626249  -2.08641622574526  54.961920626249  -2.08641622574526
54.9371914061613 -2.11262160563341  54.9371921992281 -2.11262715928642  54.9371916462945 -2.11262729538749
54.9384477074403 -2.13105527367102  54.9384587434943 -2.13105108259977  54.9384571211911 -2.13105148190769
54.9830214105886 -2.1477531391735   54.9830440131806 -2.14774832548819  54.9830437594955 -2.1477483879346
54.9340501036776 -2.10024600236184* 54.9340501036776 -2.10024600236184  54.9340501036776 -2.10024600236184
55.0059104791523 -2.1013107076275   55.0059045513926 -2.10131693880724  55.0059039378578 -2.10131708985379
54.9350555816388 -2.13941310516999  54.9350567261649 -2.13941336175938  54.9350565238902 -2.13941341154537
55.0002140023746 -2.08784613529881  55.0002144015525 -2.08784855291579  55.0002143195968 -2.0878485730927
 

The asterisk (at the end of the first dec value column) indicates there entry had neither parallax nor proper motion values in the database.  The G before the first ra column indicates the GAIA sources in the dataset footprint.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

If I enter the input values into the Web interface both for gsss and gssstest (this can be accessed even though it is not advertised), I get values in agreement with the table.  It took me a while to update this ticket as I was double-checking various items.  I deduce the EPOCH parameter is properly populated for this dataset as the results are different if the EPOCH parameter is actually used.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena I'm hopeful we are converging here! Pretty sure we are not at cross purposes. I'm looking at the test data directory:

/ifs/archive/dev/processing/hla/home/mburger/singlevisits/results_2023-09-14_gssstest/jdrz59

There is a file hst_15446_59_acs_wfc_f606w_jdrz59_gaia_sources.reg that has Gaia source positions. Those appear to have no proper motion corrections. Maybe that catalog was not generated using the same query as the pipeline actually uses? Was that file generated by your code or was it separately generated by Matthew Burger's code? If it is not related to the positions used in the pipeline, that could explain the discrepancy.

Maybe Matt can help enlighten us if you don't know where this file originates.

[stscijgbot-hstdp]

Comment by Matthew Burger on JIRA:

I did not create that file. It must be created by drizzlepac.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

@⁣rlw Sorry for the delay, but I was checking.  The file you quote is only created if the SVM_QUALITY_TESTING is turned on (generates a json file) so that harvesting can be done over as large dataset of processed data.  I have only one dataset with only a few objects, so I just created the json file.  In the table I created yesterday, I added a "G" as a prefix to the first RA column indicating the GAIA sources that were identified in the field-of-view just so we are all on the same page.  I agree that the GAIA coordinates match the NOEPOCH RA/Dec columns and do not have parallax and proper motion applied :(.  These coordinates are attributes of an important object, so I need to track down the disconnect in the coordinates with and without proper motion applied.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Because it has been shown the EPOCH parameter is used when issuing a query to the catalog service, I am going to close this ticket.  Having said this, a follow-on ticket (HLA-1129) has been created to follow through and to determine, if in fact, the proper motion corrected coordinates are used for the SVM processing.