search

chandra-mta / MTA

0 stars 0 forks source link

Acis focal fix #5

Closed william-aaron-CFA closed 1 year ago

william-aaron-CFA commented 1 year ago

This PR addressed issues #4 by updating to Ska3/flight environment, implementing lock files, unit testing, as well as reducing some dependency on MTA common functions by replacing the add_leading_zeros function with zfill as well as the text file reader.

Testing: This set of script is tested in the "/data/mta4/testACIS/Focal" test directory, with the scripts symlinked to the git version, running as mta@c3po-v. It operates on a separate copied data directory pulled from "/data/mta/Script/ACIS/Focal/Data".

Final Implementation Steps:

  1. Revert pathing from test directories to live directories
  2. Copy current live scripts to a Scripts3.8 adjacent directory
  3. Copy python, main, wrap, and perl scripts to live running
  4. Remove disk-bloat by removing backup tilde files in data directory.
  5. Remove Save2 and Save_xxx data directories.
william-aaron-CFA commented 1 year ago

Future inclusion of unit tests in create_focal_temperature_plots.py, create_week_long_plot.py, create_html_pages.py necessary.

william-aaron-CFA commented 1 year ago

This PR has addressed the following issues listed in Issues #4. General issues:

  1. Conversion to Ska3/flight environment

  2. Partial Inclusion of Unit tests

  3. Inclusion of lock files for race condition handling

  4. Remove space paradigm. full_focal_plane_data.py:

  5. Unnecessary to store the backup of the full data list for previous years. Change to removing previous years backup if the new year of focal data is being filled out.

  6. Change add_leading_zero separate function to string native method zfill().

swolk commented 1 year ago

How was this tested?

Are all 7 of these issues now addressed? 1Conversion to Ska3/flight environment 2Partial Inclusion of Unit tests 3Inclusion of lock files for race condition handling 4Remove space paradigm. 5full_focal_plane_data.py: 6 Unnecessary to store the backup of the full data list for previous years. Change to removing previous years backup if the new year of focal data is being filled out. 7Change add_leading_zero separate function to string native method zfill().

william-aaron-CFA commented 1 year ago

How was this tested?

Are all 7 of these issues now addressed? 1:Conversion to Ska3/flight environment. 2: Partial Inclusion of Unit tests. 3: Inclusion of lock files for race condition handling. 4: Remove space paradigm. 5: full_focal_plane_data.py. 6: Unnecessary to store the backup of the full data list for previous years. Change to removing previous years backup if the new year of focal data is being filled out. 7: Change add_leading_zero separate function to string native method zfill().

I tested these scripts by running them in a separate directory located at "/data/mta4/testACIS/Focal" with the house_keeping paths altering their output to run on a test bed of data and operate on a test bed of output. But I see that this approach has it's problems as it's not easily repeatable by others.

The remaining issues not addressed are:

Inclusion of unit tests for create_focal_temperature_plots.py, create_week_long_plot.py, create_html_pages.py
Convert to Astropy data fits files
Operate with arc5gl instead of pulling dumps from /dsops.
reduce dependence on mta_common_functions.

Based on the advice I received from Malgosia on Thursday, I'll be retracting this PR to better address repeatable testing and to address all these issues as separate markers to make it more clear what I have fixed and what still needs work.