sot / kadi

Chandra commands and events
https://sot.github.io/kadi
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Unexpected mismatch of obsids in starcats / observations #227

Open taldcroft opened 2 years ago

taldcroft commented 2 years ago

There is something not quite right that needs investigation:

      >>> from kadi.commands import get_starcats_as_table, get_observations
      >>> from astropy import table
      >>> start='2020:001'
      >>> stop='2021:001'
      >>> cats = get_starcats_as_table(start, stop, unique=True)
      >>> ok = np.isin(cats['type'], ['GUI', 'BOT'])
      >>> guides = cats[ok]
      >>> obss = table.Table(get_observations(start, stop))
      >>> obss = obss[~obss['starcat_date'].mask]  # keep only obs with starcat
      >>> guides = table.join(guides, obss, keys=['starcat_date'])
      >>> odd = guides2['obsid_1'] != guides2['obsid_2']
      >>> np.count_nonzero(odd)
      47
jeanconn commented 2 years ago

This particular set (if guides2 is supposed to be the second assignment of guides) looks to have some undercovers, some manual obsid transitions, and an ECS

In [36]: np.unique(guides2[odd]['obsid_2'])
Out[36]: 
<Column name='obsid_2' dtype='int64' length=6>
62647
62649
62650
62654
62655
65528
taldcroft commented 2 years ago

I think this is expected behavior. You need to use obsid as a join key. With only starcat_date it is joining (e.g.) the two obss for an undercover with the one starcat.

In [9]: cats = get_starcats(start, stop)
In [10]: np.all([cat.obsid for cat in cats]  == obss['obsid'])
Out[10]: True

Here is a slightly modified version of the original example:

In [11]:       >>> from kadi.commands import get_starcats_as_table, get_observations
    ...:       >>> from astropy import table
    ...:       >>> start='2020:001'
    ...:       >>> stop='2021:001'
    ...:       >>> cats = get_starcats_as_table(start, stop, unique=True)
    ...:       >>> ok = np.isin(cats['type'], ['GUI', 'BOT'])
    ...:       >>> guides = cats[ok]
    ...:       >>> obss = table.Table(get_observations(start, stop))
    ...:       >>> obss_sc = obss[~obss['starcat_date'].mask]  # keep only obs with starcat
    ...:       >>> guides_obs = table.join(guides, obss_sc, keys=['starcat_date', 'obsid'])
    ...: 

In [12]: len(guides)
Out[12]: 12324

In [13]: len(guides_obs)  # Did not lose any guide stars in the join
Out[13]: 12324

In [16]: set(obss_sc['obsid']) - set(guides_obs['obsid'])  # Undercovers only (?)
Out[16]: {62647, 62649, 62650, 62654, 62655}