search

lsst-uk / lsst-ir-fusion

Development of code to exploit LSST and VISTA imaging
Apache License 2.0
2 stars 0 forks source link

Updates to code to allow an SXDS rerun with the latest science pipelines #79

Closed raphaelshirley closed 7 months ago

raphaelshirley commented 7 months ago

Various changes required in order to get the Slurm machinery working with the standard pipelines.

Previously we required the gen3_workflow package but since that slurm_parsl has been integrated into the main bps functionality.

raphaelshirley commented 7 months ago

Hi @Saremi-Elham I don't understand the stuff about the conda environment in the install script. What does that do?

If this is all working I am happy for you to merge.

the only think that needs to be changed is to change the loop over ingests of images to be replaced by the the use of astro_metadata_translator and a butler ingest.

Thanks!

Saremi-Elham commented 7 months ago

Hi @raphaelshirley,

I don't understand the stuff about the conda environment in the install script.

We need to install "ndcctools" and "wq_env", as well as the science pipeline packages. I've added this step to the installation script.

the only think that needs to be changed is to change the loop over ingests of images to be replaced by the the use of astro_metadata_translator and a butler ingest.

Yes, Sure. Additionally, I'm conducting tests on the SLURM files with more nodes for a full run, and it's probable that we'll need to make some adjustments based on the results of this test. However, it's a good idea to merge the changes for now and then proceed with further work.

Thanks a lot!