search

ComputationalRadiationPhysics / clara2

Clara2 - a parallel classical radiation calculator based on Liénard-Wiechert potentials
GNU General Public License v3.0
13 stars 9 forks source link

Get rid of array job support #97

Open PrometheusPi opened 6 years ago

PrometheusPi commented 6 years ago

Dear users,

is anybody relying on the array job support? I have not used it for years and I would be very happy to rely entirely on MPI. This would allow a more lightweight code and setup routine and additionally would support getting rid of process_data and using direct MPI gather instead of storing spectra for each trajectory and disk.

The only advantage of array jobs is that they might start faster an very small clusters that are overbooked. @TheresaBruemmer Would this be the case for Maxwell? On Hypnos, Taurus and PizDaint this would not be an issue. @QJohn2017 I assume since you are using the second largest cluster in the world, it is not an issue for you too. Is this correct?

TheresaBruemmer commented 6 years ago

What/where is the array job support? I don't think I ever used it...

PrometheusPi commented 6 years ago

Array jobs are jobs that are a selection of very similar single core jobs that get an ID from the submit system. With this ID each of the jobs can perform a different task, similar to the MPI rank distributed by MPI. However no cross communication is possible.

The advantage is, that since each job is a single core job, these jobs can start as soon as a single core is available. They can increase the occupancy of a cluster, but they also put a heavy load on the scheduler.

In contrast, an MPI job is a single job on multiple cores/nodes for the scheduler. Its scheduling is not a lot of load for the scheduler. But it might take some time till enough nodes are available, so that the job can start.

TheresaBruemmer commented 6 years ago

Well, I know the concept of array jobs. I did not know what you meant by array job support, but I guess you mean the compatibility of a clara2 run with splitting it into several array jobs.

In that case, I think it is ok to just include the ./process_data in the submit.sh By that, clara2 can still be used by people who require this array job support.