The idx column in the cmds.h5 file was unfortunately sized at unsigned 16-bit int (65536). This identifies unique command parameters. The initial estimate that this would be more than enough proved to be faulty and the number exceeded 65536 with the AUG1720 loads.
Key elements are:
Changed from uint16 to int32 for the idx column.
Added code to ensure that idx=65536 is never used. This ensures that the current ska3-flight kadi version 5.2.0 runs correctly (in particular get_cmds_from_backstop) with the new files.
This PR includes a new utility script that was used to migrate the cmds.h5 as of August 8 (which did not suffer from the overflow issue) to a new version. The script contains checks that the new and previous commands are identical except for the dtype of idx.
With this PR that cmds.h5 and cmds.pkl files were updated to the present using python -m kadi.update_cmds from the git repo on this branch. They have been installed to /proj/sot/ska/data/kadi.
The footprint of the fix is small and no current flight code is impacted.
Testing
[x] Passes unit tests on linux (HEAD flight) with one workaround to ignore an expected failure in dtype comparison
[x] Passes units tests on linux (HEAD shiny)
[x] Functional testing
Functional testing
Original problem report
from kadi import commands
from kadi.commands import states
# Get commands from JUN2720 from backstop
backstop_file = "/data/acis/LoadReviews/2020/JUL2720/oflsb/CR208_1806.backstop"
cmds = commands.get_cmds_from_backstop(backstop_file)
t = states.get_states(cmds=cmds, merge_identical=True)
This produces the expected output on HEAD (kady) and GRETA (cheru).
Starcheck
Ran starcheck on HEAD on the AUG2420A loads. No process problems and gave the expected output.
Ran starcheck on HEAD on the AUG1020A loads. Outputs identical to flight except for small propagation changes.
File contents
Manually examined the cmds.pkl file and confirmed that the idx value jumps as expected and no 65535 is found.
Description
The
idxcolumn in thecmds.h5file was unfortunately sized at unsigned 16-bit int (65536). This identifies unique command parameters. The initial estimate that this would be more than enough proved to be faulty and the number exceeded 65536 with the AUG1720 loads.Key elements are:
idxcolumn.idx=65536is never used. This ensures that the current ska3-flight kadi version 5.2.0 runs correctly (in particularget_cmds_from_backstop) with the new files.This PR includes a new utility script that was used to migrate the
cmds.h5as of August 8 (which did not suffer from the overflow issue) to a new version. The script contains checks that the new and previous commands are identical except for the dtype ofidx.With this PR that
cmds.h5andcmds.pklfiles were updated to the present usingpython -m kadi.update_cmdsfrom the git repo on this branch. They have been installed to/proj/sot/ska/data/kadi.The footprint of the fix is small and no current flight code is impacted.
Testing
Functional testing
Original problem report
This produces the expected output on HEAD (kady) and GRETA (cheru).
Starcheck
File contents
Manually examined the
cmds.pklfile and confirmed that theidxvalue jumps as expected and no 65535 is found.Fixes #189