Use Python to make text report from html

jeanconn commented 3 months ago

Description

This is intended to support our arm64 build, as I have not quite gotten Perl HTML::TableExtract to build successfully on arm64 and perhaps we just don't need it.

Interface impacts

This changes newlines in the text file output of starcheck.

I think the only thing parsing the text file is the starcheck_parser code, which is functionally checked as part of this PR. Load review also uses diffs of the text file, but this PR does not introduce diffs between the vehicle and full loads.

Testing

Unit tests

[x] Mac


(ska3-flight-2024.1rc4) flame:starcheck jean$ pytest
================================================ test session starts ================================================
platform darwin -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/jean/git
configfile: pytest.ini
plugins: timeout-2.2.0, anyio-4.3.0
collected 14 items

starcheck/tests/test_state_checks.py ............. [ 92%] starcheck/tests/test_utils.py . [100%]

================================================= warnings summary ================================================== ../../miniconda3/envs/ska3-flight-2024.1rc4/lib/python3.11/site-packages/tables/node.py:251 ../../miniconda3/envs/ska3-flight-2024.1rc4/lib/python3.11/site-packages/tables/node.py:251 starcheck/starcheck/tests/test_state_checks.py::test_get_obs_man_angle[646940289.224-143.4] /Users/jean/miniconda3/envs/ska3-flight-2024.1rc4/lib/python3.11/site-packages/tables/node.py:251: DeprecationWarning: alltrue is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use all instead. self._v_objectid = self._g_open()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ========================================== 14 passed, 3 warnings in 7.76s =========================================== (ska3-flight-2024.1rc4) flame:starcheck jean$ git rev-parse HEAD a9edae5f73b558b0df4230cb5bf33edabc860786


Independent check of unit tests by [REVIEWER NAME]
- [ ] [PLATFORM]:

### Functional tests
<!-- Describe and document results of any functional tests, otherwise leave the text below -->

jeanconn-fido> pwd /proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/starcheck/starcheck-pr441 jeanconn-fido> /proj/sot/ska/bin/diff2html MAY1324B_master.txt MAY1324B_master_vehicle.txt > MAY1324B_masters_diff.html jeanconn-fido> /proj/sot/ska/bin/diff2html MAY1324B_test.txt MAY1324B_test_vehicle.txt > MAY1324B_test_diff.html

The full load to vehicle output diffs in the test outputs are consistent with the same diffs in the 'master' branch diffs:

https://icxc.cfa.harvard.edu/aspect/test_review_outputs/starcheck/starcheck-pr441/MAY1324B_masters_diff.html

https://icxc.cfa.harvard.edu/aspect/test_review_outputs/starcheck/starcheck-pr441/MAY1324B_test_diff.html

The text files as parsed by starcheck.parser show that the new output appears to work *better* with the parser in that the extra spacing by chance fixes a bug that the maneuvers of the maneuver-only observations were not getting into the dictionary of the parser in the output from master.  There's no real impact there as the starcheck parser output is intended to work to create and update a database of star catalogs (and the maneuver-only observations don't have those).

In [1]: import starcheck.parser

In [2]: test = starcheck.parser.read_starcheck('MAY1324B_test.txt')

In [3]: master = starcheck.parser.read_starcheck('MAY1324B_master.txt')

In [4]: from itertools import count

In [6]: for i, t, m in zip(count(), test, master): ...: if t != m: ...: print(i) ...: 12 19 27

In [8]: test[12] Out[8]: {'obsid': 43447, 'obs': {'obsid': 43447, 'point_ra': 331.015146, 'point_dec': 0.739498, 'point_roll': 108.991505}, 'manvrs': [{'mp_targquat_time': '2024:136:02:52:04.357', 'mp_targquat_vcdu_cnt': 1737502, 'target_Q1': 0.78721495, 'target_Q2': -0.20734763, 'target_Q3': -0.14024789, 'target_Q4': 0.56358682, 'angle_deg': 0.01, 'duration_sec': 240, 'end_date': '2024:136:02:56:10', 'instance': 0}], 'catalog': {}, 'warnings': [], 'pred_ccd_temp': None}

In [9]: master[12] Out[9]: {'obsid': 43447, 'obs': {'obsid': 43447, 'point_ra': 331.015146, 'point_dec': 0.739498, 'point_roll': 108.991505}, 'manvrs': {}, 'catalog': {}, 'warnings': [], 'pred_ccd_temp': None}

I confirmed that the text block (from the output of master) is consistent with the maneuver information printed for 43447 and parsed.

OBSID: 43447
RA, Dec, Roll (deg): 331.015146 0.739498 108.991505 BACKSTOP GUIDE_SUMM OR MANVR DOT TLR

MP_TARGQUAT at 2024:136:02:52:04.357 (VCDU count = 1737502) Q1,Q2,Q3,Q4: 0.78721495 -0.20734763 -0.14024789 0.56358682 MANVR: Angle= 0.01 deg Duration= 240 sec End= 2024:136:02:56:10

In [10]: test[19] Out[10]: {'obsid': 43445, 'obs': {'obsid': 43445, 'point_ra': 194.184776, 'point_dec': -57.404593, 'point_roll': 306.588572}, 'manvrs': [{'mp_targquat_time': '2024:137:05:55:55.852', 'mp_targquat_vcdu_cnt': 2117722, 'target_Q1': 0.37707481, 'target_Q2': 0.44414365, 'target_Q3': -0.75091653, 'target_Q4': 0.31092662, 'angle_deg': 54.89, 'duration_sec': 1392, 'end_date': '2024:137:06:19:13', 'instance': 0}], 'catalog': {}, 'warnings': [], 'pred_ccd_temp': None}

In [11]: master[19] Out[11]: {'obsid': 43445, 'obs': {'obsid': 43445, 'point_ra': 194.184776, 'point_dec': -57.404593, 'point_roll': 306.588572}, 'manvrs': {}, 'catalog': {}, 'warnings': [], 'pred_ccd_temp': None}

OBSID: 43445
RA, Dec, Roll (deg): 194.184776 -57.404593 306.588572 BACKSTOP GUIDE_SUMM OR MANVR DOT TLR

MP_TARGQUAT at 2024:137:05:55:55.852 (VCDU count = 2117722) Q1,Q2,Q3,Q4: 0.37707481 0.44414365 -0.75091653 0.31092662 MANVR: Angle= 54.89 deg Duration= 1392 sec End= 2024:137:06:19:13

In [12]: test[27] Out[12]: {'obsid': 43441, 'obs': {'obsid': 43441, 'point_ra': 331.015146, 'point_dec': 0.739498, 'point_roll': 109.521944}, 'manvrs': [{'mp_targquat_time': '2024:138:18:21:04.357', 'mp_targquat_vcdu_cnt': 2629365, 'target_Q1': 0.78981533, 'target_Q2': -0.20799461, 'target_Q3': -0.13928659, 'target_Q4': 0.55993681, 'angle_deg': 0.01, 'duration_sec': 240, 'end_date': '2024:138:18:25:10', 'instance': 0}], 'catalog': {}, 'warnings': [], 'pred_ccd_temp': None}

In [13]: master[27] Out[13]: {'obsid': 43441, 'obs': {'obsid': 43441, 'point_ra': 331.015146, 'point_dec': 0.739498, 'point_roll': 109.521944}, 'manvrs': {}, 'catalog': {}, 'warnings': [], 'pred_ccd_temp': None}

OBSID: 43441
RA, Dec, Roll (deg): 331.015146 0.739498 109.521944 BACKSTOP GUIDE_SUMM OR MANVR DOT TLR

MP_TARGQUAT at 2024:138:18:21:04.357 (VCDU count = 2629365) Q1,Q2,Q3,Q4: 0.78981533 -0.20799461 -0.13928659 0.55993681 MANVR: Angle= 0.01 deg Duration= 240 sec End= 2024:138:18:25:10



And everything else is identical.

taldcroft commented 3 months ago

This just needs documentation that the output for MAY1324B is unchanged from the last commit.

jeanconn commented 3 months ago

Right, I just actually need to do that first 😄

jeanconn commented 3 months ago

I reran the output at the previous "last commit" (because the agasc supplement has changed) and confirmed just expected diffs.

ska3-jeanconn-fido> diff MAY1324B_test.txt MAY1324B_rtest.txt
1,2c1,2
<  ------------  Starcheck 14.8.2.dev1+ga9edae5    -----------------
<  Run on Mon May 13 12:05:54 EDT 2024 by jeanconn from fido.cfa.harvard.edu
---
>  ------------  Starcheck 14.8.2.dev4+g21ef617    -----------------
>  Run on Mon May 13 12:15:22 EDT 2024 by jeanconn from fido.cfa.harvard.edu

sot / starcheck