search

LSSTDESC / ceci

Experimental pipeline prototype software
BSD 3-Clause "New" or "Revised" License
13 stars 9 forks source link

Parallelism on batch systems #4

Closed joezuntz closed 6 years ago

joezuntz commented 6 years ago

The current setup handles parallelism alright on test machine but not when submitting to clusters or supercomputers via batch systems.

As I understand it we currently need to dynamically create a new site configuration for each size of MPI job, but this could be wrong.

joezuntz commented 6 years ago

Need to ask the parsl people about this!

joezuntz commented 6 years ago

There is now a parsl issue looking at this: https://github.com/Parsl/parsl/issues/224

Given how impressively fast parsl development has generally progressing been it will probably be best to wait for that to be resolved.

joezuntz commented 6 years ago

This is now mostly addressed in branch features/parallel. Still querying on the specific behaviour regarding wall times, but the code now works on cori.

joezuntz commented 6 years ago

features/parallel is merged in