search

CRPropa / CRPropa3

CRPropa is a public astrophysical simulation framework for propagating extraterrestrial ultra-high energy particles. https://crpropa.github.io/CRPropa3/
https://crpropa.desy.de
GNU General Public License v3.0
68 stars 67 forks source link

Runnig CRpropa on cluster #369

Closed PrantikS closed 2 years ago

PrantikS commented 2 years ago

Hi,

I wish to run CRPropa on cluster using multiple nodes. CRPropa can use all the processors available in a single node as it has OpenMP support. But it can not utilise processors across different nodes (perhaps, it requires MPI support). Therefore, I want to distribute multiple copies of CRPropa simulation code in different nodes using a single JOB script. For example, I have 3 cluster nodes: node1, node2, node3 and I want to run 3 CRPropa codes (say Sim1.py, Sim2.py, Sim3.py) in these nodes simultaneously in such a way that Sim1.py goes to node1, Sim2.py goes to node2 etc.

The job submission is done using SLURM script. I tried a few SLURM scripts to run CRPropa in the way that I have described above, but it didn't work. Is it possible to run CRPropa in this way? Can you help me to figure out a SLURM script which can fulfil my purpose. Any alternate suggestions would also be very helpful.

Thank you!

TobiasWinchen commented 2 years ago

CRPropa has been used on clusters by numerous people so far, and I personally submitted CRPropa jobs with SLURM without any issues. Please provide further details on the problems you encounter and what you are trying to do. Please make also sure that the issue you encounter is specific to CRPropa by running a minimal example code such as

sim1.py

print('Sim 1')

sim2.py

print('Sim 2')

etc.

on the cluster nodes.

PrantikS commented 2 years ago

Hi, Here is the code:

sim1.py

`from crpropa import *

photons= True neutrinos=True electrons= True

set up random turbulent field vgrid = Grid3f(Vector3d(0), 512, 30 kpc) initTurbulence(vgrid, 8 nG, 60 kpc, 800 kpc, -11. / 3, 42) Bfield = MagneticFieldGrid(vgrid)

simulation setup sim = ModuleList() sim.add(PropagationCK(Bfield)) sim.add(FutureRedshift()) sim.add(PhotoPionProduction(CMB(), photons, neutrinos, electrons)) sim.add(ElectronPairProduction(CMB())) sim.add(ElectronPairProduction(IRB_Finke10())) sim.add(EMPairProduction(URB_Protheroe96(),True)) sim.add(EMInverseComptonScattering(CMB(),True)) sim.add(EMDoublePairProduction(URB_Protheroe96(),True)) sim.add(EMInverseComptonScattering(URB_Protheroe96(),True)) sim.add(EMTripletPairProduction(URB_Protheroe96(),True)) sim.add(MinimumEnergy(1 * EeV))

periodic boundaries extent = 512 30 kpc # size of the magnetic field grid sim.add(PeriodicBox(Vector3d(-extent), Vector3d(2 * extent)))

define the observer obs = Observer() obs.add(ObserverSmallSphere(Vector3d(0, 0, 0), 0.5 * Mpc)) obs.add(ObserverRedshiftWindow(-0.05, 0.05)) output = TextOutput(output1.txt, Output.Event3D) output.disableAll() # disable everything to start from scratch output.enable(Output.CurrentEnergyColumn) # current energy output.enable(Output.CurrentIdColumn) # current particle type output.enable(Output.SourceEnergyColumn) # current energy output.enable(Output.SourceIdColumn) # current particle type

obs.onDetection(output) sim.add(obs)

define the source(s) source = Source() source.add(SourcePosition(Vector3d(50, 0, 0) Mpc)) source.add(SourceIsotropicEmission()) source.add(SourceParticleType(nucleusId(1, 1))) source.add(SourcePowerLawSpectrum(1 EeV, 10000 * EeV, -2.0)) source.add(SourceRedshiftEvolution(1.5, 0.001, 3))

run simulation sim.setShowProgress(True) sim.run(source, 5000000) `

sim2.py, sim3.py, etc are the same copies of sim1.py with different output file names. I am trying to run sim1.py, sim2.py etc on different nodes simultaneously.

Suppose, I have five CRPropa codes sim1.py ....sim5.py

I am using a SLURM script as below to run these five codes

`#!/bin/bash

SBATCH -J CRPropa1

SBATCH --nodes=5

SBATCH --ntasks=1

SBATCH --error=job.%J.err

SBATCH --output=job.%J.out

SBATCH --time=50:00:00

for i in {1..5} do python sim$i.py done

` I want to send the codes sim1.py ........sim5.py to the 5 different nodes that I have chosen. But this way it doesn't work. Perhaps the issue is with the SLURM script.

I am actually new to cluster computing. So, if you can provide some information on how to do this, it would be really helpful!

TobiasWinchen commented 2 years ago

The issue is in your slurm script, you are executing all scripts on all nodes. A minimum working example (although quite tedious) would be to create 5 slurm scripts that submit one job to one node. Please ask you IT Team respectively cluster administrator for an introduction into your system or look for online tutorials.

PrantikS commented 2 years ago

Thanks a lot! Just to clarify, are you talking about creating JOB array?