Cache starcat and other goodies

taldcroft commented 2 years ago

Description

This is hodge-podge PR that makes improvements related to getting star catalogs from the commands archive.

Adds get_starcats_as_table function
Changes API on get_observations and get_starcats. First three args are now start=None, stop=None, *, obsid=None, .... This is more consistent with signatures for other methods in kadi commands.
Adds file-based caching of star catalog data to greatly improve performance.
Adds as_dict option to get_starcats to allow getting catalogs as plain Python dict instead of converting to ACATable objects. This is another performance-driven feature.
Changes API to update start/stop date filtering in get_observations (and by proxy get_starcats) so the filtering is inclusive. That means any observation where obs_start / obs_stop intersects start / stop will be included. Previously the filtering was just start <= obs_start <= stop.
Better exception handling (#228) related to web access

Fixes #228

Interface impacts

Two API changes as noted above.

Testing

Unit tests

[x] Mac

Independent check of unit tests by Jean

[x] Mac and Linux

Functional tests

Caching control

After removing ~/.kadi/starcats.db, I ran the code below and confirmed that the cache was NOT created, as expected:

from kadi.commands import get_starcats, conf
with conf.set_temp('cache_starcats', False):
    starcats = get_starcats(start='2020:001', stop='2020:100')

Then I did the same starcats = ... command but with caching enabled (the default) and confirmed that the file was created and that the expected performance improvement was seen on subsequent queries.

HTTP exceptions

Testing of d1d30fd consisted of:

Change my ~/.netrc to force an authentication error. Confirmed that I get a 401 authentication error when trying get_observations(obsid=8008).
Change the code temporarily to force all of the directory download requests to fail, thus falling through to the line which raises ValueError(f'No loads found in {lookback} days'). This worked as expected.

taldcroft commented 2 years ago

@jeanconn @javierggt - this is ready (I hope!) for final review.

taldcroft commented 2 years ago

@jeanconn - I think the comments you made have been addressed.

sot / kadi