Configure kadi logger globally + other refactoring

[taldcroft]

Description

Fixes the issue noted on slack (#missionplanning "SOT MP run of starcheck did not generate the expected output files for the OCT1722B products"), that get_dither_kadi_state would run before KADI_SCENARIO=flight was set, causing HEAD starcheck to still have a network dependence for SOT MP's run of the tool.

In the course of implementing the fix, some refactoring and rework of existing code was useful.

Interface impacts

Testing

Unit tests

• [x] No unit tests

Functional tests

Disabled occweb access and confirmed that master fails to use 'flight' scenario:

Traceback (most recent call last):
  File "/proj/sot/ska/jeanproj/git/starcheck/starcheck/utils.py", line 26, in wrapper
    return func(*args, **kwargs)
  File "/proj/sot/ska/jeanproj/git/starcheck/starcheck/utils.py", line 100, in get_dither_kadi_state
    state = states.get_continuity(date, cols)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/states.py", line 1697, in get_continuity
    cmds = commands.get_cmds(start, stop, scenario=scenario)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands.py", line 54, in get_cmds
    cmds = get_cmds_(start=start, stop=stop, inclusive_stop=inclusive_stop, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands_v2.py", line 216, in get_cmds
    cmds_recent = update_archive_and_get_cmds_recent(
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands_v2.py", line 312, in update_archive_and_get_cmds_recent
    loads = update_loads(scenario, cmd_events=cmd_events, lookback=lookback, stop=stop)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands_v2.py", line 911, in update_loads
    contents = occweb.get_occweb_dir(dir_year_month)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/occweb.py", line 322, in get_occweb_dir
    html = get_occweb_page(path, timeout=timeout, cache=cache)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/occweb.py", line 290, in get_occweb_page
    req.raise_for_status()  # raise exception if not 200
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://occweb.cfa.harvard.edu/occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/SEP
HTTPError: 401 Client Error: Unauthorized for url: https://occweb.cfa.harvard.edu/occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/SEP at line 26
 at ./starcheck/src/starcheck.pl line 37.
    main::__ANON__("HTTPError: 401 Client Error: Unauthorized for url: https://oc"...) called at (eval 184) line 3
    main::get_dither_kadi_state("__main__", "get_dither_kadi_state", "2022:289:13:17:00.000") called at ./starcheck/src/starcheck.pl line 274

Confirmed that this PR (with the same disabled .netrc etc) works and that KADI_SCENARIO is set to flight before/for all kadi command queries.

ska3-jeanconn-fido> ./sandbox_starcheck -dir /data/mpcrit1/mplogs/2022/OCT1722/oflsb/ -out ob_test
Using backstop file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//CR289_1303.backstop
Using guide summary file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/mgOCT1722B.sum
Using OR file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/or/OCT1722_B.or
Using maneuver file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/mmOCT1722B.sum
Using DOT file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/mdOCT1722B.dot
Using mech check file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//output/TEST_mechcheck.txt
Using fidsel file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/FIDSEL.txt
Using dither file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/DITHER.txt
Using radmon file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/RADMON.txt
Using simtrans file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/SIMTRANS.txt
Using simfocus file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/SIMFOCUS.txt
Using attitude file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/ATTITUDE.txt
Using characteristics file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/ode/characteristics/CHARACTERIS_18JAN21
Using aimpoint file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//output/OCT1722B_dynamical_offsets.txt
Using config file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/characteristics.yaml
Using odb file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/fid_CHARACTERISTICS
Using agasc_file file /proj/sot/ska3/flight/data/agasc/proseco_agasc_1p7.h5
Using manerr file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//output/OCT1722B_ManErr.txt
Using processing summary file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/msOCT1722B.sum
Using TLR file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//CR289_1303.tlr
Using banned_agasc file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/agasc.bad
Using bad_pixel file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/ACABadPixels
Using acq_star_rdb file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/bad_acq_stars.rdb
Using gui_star_rdb file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/bad_gui_stars.rdb
Getting commands from archive only
Loaded /proj/sot/ska3/flight/data/kadi/cmds2.h5 with 1431351 commands
Loaded /proj/sot/ska3/flight/data/kadi/cmds2.pkl with 147968 pars
Read 258 ACA bad pixels from /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/ACABadPixels
Read 46 bad AGASC IDs from /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/agasc.bad
#####################################################################
# calc_ccd_temps run at Tue Oct 18 10:36:10 2022 by jeanconn
# Continuity run_start_time = 2022:291:14:36:09.564
# calc_ccd_temps version = 13.16.1.dev3+g75cba91
# chandra_models version = 3.43
# kadi version = 7.0.2
#####################################################################

Using backstop file /data/mpcrit1/mplogs/2022/OCT1722/oflsb/CR289_1303.backstop
Found 2274 backstop commands between 2022:289:13:17:00.000 and 2022:297:11:45:00.000
RLTT = 2022:289:13:20:00.000
sched_stop = 2022:297:11:45:00.000
Fetching telemetry between 2022:288:13:17:00.000 and 2022:289:13:17:00.000
Getting commands from archive only
Getting commands from archive only
Calculating ACA thermal model
Propagation initial time and ACA: 2022:289:13:02:46.816 -8.76
Making temperature check plots
Writing plot file ob_test/ccd_temperature.png
Checking star catalog for obsid 45165
Checking star catalog for obsid 45164
Checking star catalog for obsid 45162
Checking star catalog for obsid 45161
Checking star catalog for obsid 45160
Checking star catalog for obsid 45159
Checking star catalog for obsid 26641
Checking star catalog for obsid 27505
Checking star catalog for obsid 25244
Checking star catalog for obsid 27517
Checking star catalog for obsid 27512
Checking star catalog for obsid 27514
Checking star catalog for obsid 26031
Checking star catalog for obsid 27271
Checking star catalog for obsid 27305
Checking star catalog for obsid 27306
Checking star catalog for obsid 27307
Checking star catalog for obsid 27308
Checking star catalog for obsid 27309
Checking star catalog for obsid 27310
Checking star catalog for obsid 27311
Checking star catalog for obsid 45158
Checking star catalog for obsid 45157
Checking star catalog for obsid 45156
Checking star catalog for obsid 45154
Checking star catalog for obsid 45153
Checking star catalog for obsid 45152
Checking star catalog for obsid 45151
Checking star catalog for obsid 25805
Checking star catalog for obsid 25194
Checking star catalog for obsid 27515
Checking star catalog for obsid 27511
Checking star catalog for obsid 25806
Checking star catalog for obsid 27506
Checking star catalog for obsid 25294
Checking star catalog for obsid 25310
Checking star catalog for obsid 27509
Checking star catalog for obsid 27272
Checking star catalog for obsid 27273
Checking star catalog for obsid 27312
Checking star catalog for obsid 27313
Checking star catalog for obsid 27314
Checking star catalog for obsid 27315
Checking star catalog for obsid 27316
Checking star catalog for obsid 27337
Checking star catalog for obsid 27338
Checking star catalog for obsid 27339
Checking star catalog for obsid 45150
Checking star catalog for obsid 45149
Checking star catalog for obsid 45148
Checking star catalog for obsid 45146
Checking star catalog for obsid 45145
Checking star catalog for obsid 45144
Checking star catalog for obsid 45143
Checking star catalog for obsid 25111
Checking star catalog for obsid 27510
Checking star catalog for obsid 23863
Checking star catalog for obsid 27516
Checking star catalog for obsid 27508
Checking star catalog for obsid 27513
Checking star catalog for obsid 27507
Checking star catalog for obsid 27482
Checking star catalog for obsid 25550
Checking star catalog for obsid 27317
Checking star catalog for obsid 27318
Checking star catalog for obsid 27319
Checking star catalog for obsid 27320
Checking star catalog for obsid 27321
Checking star catalog for obsid 24407
Checking star catalog for obsid 45142
Getting commands from archive only
Wrote HTML report to ob_test.html
Wrote text report to ob_test.txt

Testing of 403a1d6

By default the output does not include any kadi logging output. Confirmed this with:

./sandbox_starcheck -dir ~/ska/data/mpcrit1/mplogs/2022/OCT1722/oflsb/ -out ob_test 

Also ran with -verbose=0 and got the same results. Then ran with -verbose=2 and got kadi DEBUG output:

...
Using acq_star_rdb file /Users/aldcroft/git/starcheck/starcheck/data/bad_acq_stars.rdb
Using gui_star_rdb file /Users/aldcroft/git/starcheck/starcheck/data/bad_gui_stars.rdb
Getting cmd_events from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv
Writing 53 cmd_events to /Users/aldcroft/.kadi/cmd_events.csv
Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/OCT with cache=False
Already have /Users/aldcroft/.kadi/loads/OCT1022A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT1722B.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2122A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2422A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2422B.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2722A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT3122A.pkl.gz
Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/NOV with cache=False
No OCCweb directory for FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/NOV
Including loads OCT1022A, OCT1722B, OCT2122A, OCT2422A, OCT2422B, OCT2722A, OCT3122A
Load OCT1022A has 2143 commands with RLTT=2022:283:02:40:34.206
Load OCT1722B has 2274 commands with RLTT=2022:289:13:20:00.000
... and so on

Previous behavior reported by @jeanconn, NO LONGER accurate

The new STDERR output has "Getting commands from archive only" whenever a call was made to kadi commands. Seems fine. The first call also shows when the commands were loaded (happens once)

Loaded /proj/sot/ska3/flight/data/kadi/cmds2.h5 with 1431351 commands
Loaded /proj/sot/ska3/flight/data/kadi/cmds2.pkl with 147968 pars

This is fine as well.

[jeanconn]

What was the hidden dependence? Or, equivalently, how does this fix it?

[taldcroft]

The call to kadi get_cmds to set the dither history. So not exactly "hidden", but we both missed this in my patch that just addressed the call in calc_ccd_temps.

[taldcroft]

And the fix is to set the environment variable early in the perl script before anything else happens.

[jeanconn]

Makes sense. Thanks!

[taldcroft]

@jeanconn - just a reminder here that you should take over the functional testing of this, which I can then review.

[jeanconn]

@taldcroft I don't think I can request a review on it from you, but is that what you had in mind for functional testing or would you like some other variations?

[jeanconn]

@taldcroft figured I'd check once more about this now that you are back.

[taldcroft]

@jeanconn - I have reviewed your functional testing and it looks fine. I pushed another commit so I'll re-request a final review from you. I'll update the functional test description momentarily.