Closed vlnistor closed 3 months ago
When it comes to trying to add feedback on creating a WorkUnit
from an ImageCollection
I couldn't get an example to work:
discrete_piles_e2e.ipynb
contains an example but requires the lsst
stacktest_imagecollection.py
has a set of mock ImageCollection
s but they throw when calling toWorkUnit()
on themI played with the reproject_demo
notebook. I added a time.sleep(3)
inside _reproject_images
. For a work unit with 3 unique_obstimes_indices
, and MAX_PROCESSES
set to 1, the progress bar did 3 hops every 3 seconds (as expected). When you set the MAX_PROCESSES
to 8 (above the number of parallell jobs) the progress bars hops once (after 3 seconds) as there were enough threads to do the jobs in parallel (again, as expected).
stderror
for tqdm
output to appear (will that be an issue?). It looks similar to thisReprojecting: 0%| | 0/3 [00:00]
Reprojecting: 100%|██████████| 3/3 [00:00]
I think the line numbers may have gotten shifted around, sorry, there should also be tqdm
on this line as well in work_unit.py, the loop
for i in range(num_images):
sci_hdu = hdul[f"SCI_{i}"]
# Read in the layered image from different extensions.
img = LayeredImage(
sci_hdu.data.astype(np.single),
hdul[f"VAR_{i}"].data.astype(np.single),
hdul[f"MSK_{i}"].data.astype(np.single),
PSF(hdul[f"PSF_{i}"].data),
sci_hdu.header["MJD"],
)
which is the one that actually loads in the images. Everything else looks good!
@maxwest-uw that should all be pushed now. Let me know if anything else
This PR aims to tackle #641. I use
tqdm
package for feedbackWorkUnit
fromfits
file