Draft of multiprocessing work.

[bmiszalski]

The base code here was sourced from Tim Davies' branch tdv-PR44-fixed-merge-conflict. I've made this PR so the code may be inspected / reviewed.

I've added the following:

• multiprocessing for wavelength calibration (each slice is a separate process)
• multiprocessing to allow blue and red data to be simultaneously reduced. The required a gargs (global args) dict to be setup for the recipes.
• multiprocessing for run_slitlet_mef and run_obs_coadd
• a decorator wifes_recipe to print the name of each recipe and its duration either side of the recipe output

This allows the sample dataset classical_20230922 to be reduced in ~5 min on a fast multicore machine.

Todo:

• Perhaps use tqdm and contextlib to capture the output of all recipes and redirect to a logfile for blue and red arms separately.

[bmiszalski]

I've updated the branch with the following added:

• Capture all the output of the recipes using contextlib and redirect them to data_products/intermediate/{arm}/{arm}.log. I experimented with tqdm to make progress bars but this is unstable in the multiprocessing environment (it raises exceptions when trying to interact with the Python processes), so in the end I took a simple approach.
• Print the progress of the reductions (see below for sample output)
• Adjusted multiprocessing to have some limit on number of processes.

Sample output:

> python3 reduce_data.py . --reduce-both
Processing data in directory: /code/WiFeS/classical_20230922/
Running blue/run_overscan_sub (1/23)
Running red/run_overscan_sub (1/26)
Running red/run_bpm_repair (2/26)
Running blue/run_bpm_repair (2/23)
Running red/run_superbias (3/26)
Running blue/run_superbias (3/23)
Running red/run_bias_sub (4/26)
Running blue/run_bias_sub (4/23)
Running red/run_superflat (5/26)
Running blue/run_superflat (5/23)
Running red/run_superflat (6/26)
Running red/run_slitlet_profile (7/26)
Running red/run_flat_cleanup (8/26)
Running blue/run_superflat (6/23)
Running red/run_superflat_mef (9/26)
Running blue/run_slitlet_profile (7/23)
Running blue/run_flat_cleanup (8/23)
Running blue/run_superflat_mef (9/23)
Running red/run_superflat_mef (10/26)
Running red/run_slitlet_mef (11/26)
Running blue/run_superflat_mef (10/23)
Running red/run_wave_soln (12/26)
Running blue/run_slitlet_mef (11/23)
Running blue/run_wave_soln (12/23)
Running red/run_wire_soln (13/26)
Running red/run_flat_response (14/26)
Running red/run_cosmic_rays (15/26)
Running red/run_sky_sub (16/26)
Running red/run_obs_coadd (17/26)
Running blue/run_wire_soln (13/23)
Running blue/run_flat_response (14/23)
Running blue/run_cosmic_rays (15/23)
Running red/run_flatfield (18/26)
Running red/run_cube_gen (19/26)
Running blue/run_sky_sub (16/23)
Running blue/run_obs_coadd (17/23)
Running blue/run_flatfield (18/23)
Running blue/run_cube_gen (19/23)
Running red/run_extract_stars (20/26)
Running red/run_derive_calib (21/26)
Running red/run_flux_calib (22/26)
Running red/run_extract_stars (23/26)
Running red/run_derive_telluric (24/26)
Running red/run_telluric_corr (25/26)
Running red/run_save_3dcube (26/26)
Running blue/run_extract_stars (20/23)
Running blue/run_derive_calib (21/23)
Running blue/run_flux_calib (22/23)
Running blue/run_save_3dcube (23/23)
Saving extracted spectra for OBK-525056-WiFeS-Blue-UT20230922T121452-0.cube.fits
Saving extracted spectra for OBK-525056-WiFeS-Red--UT20230922T122551-3.cube.fits
All done in 325.7 seconds.

[timothyfrankdavies]

As @CMartinezLombilla shared via email, this seems to be forked from an Oct 2021 commit of pywifes: 7e0c34dcf73b63fb6a2779998c92d22e16f4eed2

Have you definitely used the correct branch?

If so, to fix it, and seeing as you've got a fork, you'll need to fetch latest from the original repo (maybe named upstream or origin), and then merge the automation branch:

git fetch upstream
git merge origin/automation

That gives a number of merge conflicts. I'll have a quick try at it myself in another branch, I'll let you know how I go.

Update

I haven't thoroughly looked through reduce_data.py or wifes_wsol.py, but it looks like your version actually was up to date, the merge history just didn't carry over.

So the merge conflicts can be resolved by accepting all of your changes, and then reviewing the result.

I'll give it a try myself, but I might not have permissions to update the branch on your fork. If so, I'll make my own branch & PR just so I can start the review.

2nd Update

My mistake, you were up to date with tdv-PR44-fixed-merge-conflict, which doesn't have the 10th May commit to the automation branch.

I'll get that fixed in https://github.com/PyWiFeS/pipeline/pull/59

[timothyfrankdavies]

Started a review over in https://github.com/PyWiFeS/pipeline/pull/59, @bmiszalski . I don't think I can tag/ping you there as you're not listed as a developer in pywifes. I'll add more on Monday.