Remove timelines / cmd_states.db3 from mica

taldcroft commented 1 year ago

Description

This removes the cmd_states.db3 (and timelines) dependence from mica.

I'm not sure if any of these changes impact or break processing that is not covered in unit tests. This definitely needs close review by @jeanconn.

Interface impacts

It's possible that some old (pre-2003) queries may give different results.
The status fields have been reduced/simplified and might give different results from the previous version.
The catalog column of the Table from get_monitor_windows() is now always None while previously it contained the starcheck catalog dict (e.g. from get_starcheck_catalog_at_date()).

Testing

Unit tests

[x] Mac
[x] Linux (kady)

Independent check of unit tests by Jean

[X] Linux (fido)

Functional tests

None but many new unit tests.

taldcroft commented 1 year ago

@jeanconn - I think I have addressed the comments. In addition I ran the unit tests on kady and all passed.

jeanconn commented 1 year ago

Also somewhat fundamentally, the obsid from kadi observations does not match starcheck obsids for the vehicle-only portions of schedules. I added a helper function to just make it work, but this is slower and uglier still. But since you started this as very much an incremental patch to mica to strip out timelines, maybe it makes sense. So let me know what you think makes sense for c052566

jeanconn commented 1 year ago

It might make more sense to use the kadi observations more directly as a time -> products mapping and avoid the double lookups.

taldcroft commented 1 year ago

@jeanconn - I think the new function is good now and well-tested so let's go with that. From my perspective this PR is good to go so you just need to finish the review.

jeanconn commented 1 year ago

This works on flight.

In [1]: import mica.starcheck

In [2]: mica.starcheck.get_starcheck_catalog_at_date('2023:099:04:21:40.719')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 mica.starcheck.get_starcheck_catalog_at_date('2023:099:04:21:40.719')

File /proj/sot/ska/jeanproj/git/mica/mica/starcheck/starcheck.py:203, in get_starcheck_catalog_at_date(date, starcheck_db)
    197 mp_dir = load_name_to_mp_dir(obs["source"])
    199 # Kadi observations are labeled by the commanded obsid.  Starcheck catalogs
    200 # are labeled by the planned obsid used in the starcheck output.  For observations
    201 # after SCS107 with SOSA, the kadi observation obsids and starcheck obsids don't match.
    202 # Use a helper function to line these up by starcat time.
--> 203 obsid = get_starcheck_db_obsid(obs["starcat_date"], mp_dir)
    205 # Use the obsid and the known products directory to use the more generic
    206 # get_starcheck_catalog to fetch the right one from the database
    207 cat_info = get_starcheck_catalog(obsid, mp_dir=mp_dir, starcheck_db=starcheck_db)

File /proj/sot/ska/jeanproj/git/mica/mica/starcheck/starcheck.py:315, in get_starcheck_db_obsid(mp_starcat_time, mp_dir, starcheck_db)
    307 result = starcheck_db.fetchone(
    308     "SELECT starcheck_obs.obsid FROM starcheck_obs "
    309     "INNER JOIN starcheck_id "
   (...)
    312     f"AND starcheck_obs.mp_starcat_time = '{mp_starcat_time}'"
    313 )
    314 if result is None:
--> 315     raise ValueError(f"no starcheck entry for {mp_starcat_time=} and {mp_dir=}")
    317 obsid = result["obsid"]
    318 return obsid

ValueError: no starcheck entry for mp_starcat_time=masked and mp_dir='/2023/APR0323/oflsa/'

jeanconn commented 1 year ago

I suppose technically I should have said that "doesn't throw an unhandled exception" on flight. I'm not sure if flight really has the right answer. It looks like maybe kadi observations use a masked starcat date for the gyro holds? That make sense. For this application maybe it makes sense to just use that obsid instead of the date to return an empty-ish starcheck entry?

taldcroft commented 1 year ago

I noticed today this additional interface change, which I have put into the top description:

The catalog column of the Table from get_monitor_windows() is now always filled with None while previously it contained the starcheck catalog dict (e.g. from get_starcheck_catalog_at_date()).

sot / mica