Allow outputting all keys in reduce_states

[taldcroft]

Description

This adds a keyword arg all_keys to the reduce_states function so that when True all the original state keys are output instead of just those corresponding to the reduction keys.

Search "issue I'm running into now is replacing the old commanded states version" in ska-dev on slack for context.

Interface impacts

Adds a keyword arg to the function signature but no change by default.

Testing

Unit tests

• [x] Mac

Independent check of unit tests by [REVIEWER NAME]

• [ ] [PLATFORM]:

Functional tests

Using the original example from @matthewdahmer :

In [1]: from kadi.commands import states
   ...: tstart = '2022:001:00:00:00'
   ...: tstop = '2022:073:00:00:00'
   ...: keys = ['pitch', 'off_nom_roll', 'ccd_count', 'fep_count', 'clocking', 'vid_board', 'pcad_mode', 'simpos']
   ...: state_data = states.get_states(tstart, tstop, state_keys=keys, merge_identical=True)

In [2]: state_data_reduce = states.reduce_states(state_data, ['pcad_mode'], all_keys=True, merge_identical=True)

In [3]: state_data_reduce
Out[3]: 
<Table length=955>
      datestart              datestop           tstart        tstop     ... vid_board pcad_mode simpos trans_keys
        str21                 str21            float64       float64    ...   int64      str4   int64    object  
--------------------- --------------------- ------------- ------------- ... --------- --------- ------ ----------
2022:001:00:00:00.000 2022:001:05:21:28.604 757382469.184 757401757.788 ...         1      NPNT  75624           
2022:001:05:21:28.604 2022:001:05:48:44.808 757401757.788 757403393.992 ...         1      NMAN -99616  pcad_mode
                  ...                   ...           ...           ... ...       ...       ...    ...        ...
2022:072:19:45:45.624 2022:072:20:18:24.805 763588014.808 763589973.989 ...         0      NMAN -99616  pcad_mode
2022:072:20:18:24.805 2022:073:00:00:00.000 763589973.989 763603269.184 ...         0      NPNT -99616  pcad_mode

In [4]: state_data.info
Out[4]: 
<Table length=5432>
    name      dtype  format
------------ ------- ------
   datestart   str21       
    datestop   str21       
      tstart float64    .3f
       tstop float64    .3f
       pitch float64       
off_nom_roll float64       
   ccd_count   int64       
   fep_count   int64       
    clocking   int64       
   vid_board   int64       
   pcad_mode    str4       
      simpos   int64       
  trans_keys  object   

[matthewdahmer]

Here is the code I used to test this PR:

import numpy as np
from kadi.commands import states

tstart = '2020:001:00:00:00'
tstop = '2022:073:00:00:00'
keys = ['pitch', 'off_nom_roll', 'ccd_count', 'fep_count', 'clocking', 'vid_board', 'pcad_mode', 'simpos']
state_data = states.get_states(tstart, tstop, state_keys=keys, merge_identical=True)
state_data_reduce = states.reduce_states(state_data, ['pcad_mode'], all_keys=True, merge_identical=True)

# Convert PCAD States to numbered values (e.g. NPNT --> 0, NMAN --> 1, etc.)
pcad_numbered_states = np.zeros_like(state_data['pcad_mode'])
for n, state in enumerate(set(state_data['pcad_mode'])):
    pcad_numbered_states[state_data['pcad_mode'] == state] = n

# Find indices for PCAD state changes (first row for each change)
state_diff = np.diff(pcad_numbered_states.astype(int))
ind = np.abs(np.concatenate(([1,], state_diff))) != 0

# Ensure the data in the reduced states matches the first row for each change in the original table
assert np.all(state_data_reduce['tstart'] == state_data['tstart'][ind])
assert np.all(state_data_reduce['pitch'] == state_data['pitch'][ind])
assert np.all(state_data_reduce['off_nom_roll'] == state_data['off_nom_roll'][ind])
assert np.all(state_data_reduce['ccd_count'] == state_data['ccd_count'][ind])
assert np.all(state_data_reduce['fep_count'] == state_data['fep_count'][ind])
assert np.all(state_data_reduce['clocking'] == state_data['clocking'][ind])
assert np.all(state_data_reduce['vid_board'] == state_data['vid_board'][ind])
assert np.all(state_data_reduce['pcad_mode'] == state_data['pcad_mode'][ind])
assert np.all(state_data_reduce['simpos'] == state_data['simpos'][ind])

# Find indices for each row before PCAD state changes (last states for each group)
ind = np.abs(np.concatenate((state_diff, [1,]))) != 0

# Ensure the ending time for each reduced group of states is the last time for that group
assert np.all(state_data_reduce['tstop'] == state_data['tstop'][ind])