search

lsst-uk / ukdac1

Work towards UKDAC1 deployment of the UK's LSST Data Access Centre
0 stars 0 forks source link

SXDS benchmarking #137

Open markgbeckett opened 2 years ago

markgbeckett commented 2 years ago

Re-run the SXDS pipeline processing example on CSD3 and check outputs are consistent with production run (done previously)

markgbeckett commented 1 year ago

Dominic experimented with visualisation of Quantum Graphs from Parsl, and plotting efficiency of pipeline tasks. These are useful diagnostic tools.

markgbeckett commented 1 year ago

Still investigating job efficiency problem encountered by Dominic. Issue is in single-frame processing step of pipeline, which should be relatively straightforward. Dominic has been getting to grips with the Parsl executor and how it interacts with Slurm. Dominic has also tested several different versions of the pipeline. No obvious source of issue identified so far.

markgbeckett commented 1 year ago

Will benchmark once has newest release of Pipeline and is using Ctrl-bps-parsl.

markgbeckett commented 1 year ago

Dominic has switched from Gen 3 workflow to Ctrl/ BPS, which has produced a significant improvement in performance (up to 25%). Still seeing a number of nodes idle during processing. Will look to update to newest version of Ctrl/ BPS and batch system. Also, may be seeing issue with job scheduling.

markgbeckett commented 1 year ago

@dsloanm-epcc reported that new version of Pipeline and Ctrl-BPS is close to setup -- new ObsVista is installed and ingest is working -- though name-space change causes problem with single-visit-image processing. Raphael noted that Tim J had requested a change to namespace for ObsVISTA, to avoid misassumption that ObsVISTA is a Rubin-supported package.