Closed PrantikS closed 2 years ago
CRPropa has been used on clusters by numerous people so far, and I personally submitted CRPropa jobs with SLURM without any issues. Please provide further details on the problems you encounter and what you are trying to do. Please make also sure that the issue you encounter is specific to CRPropa by running a minimal example code such as
sim1.py
print('Sim 1')
sim2.py
print('Sim 2')
etc.
on the cluster nodes.
Hi, Here is the code:
sim1.py
`from crpropa import *
photons= True neutrinos=True electrons= True
set up random turbulent field vgrid = Grid3f(Vector3d(0), 512, 30 kpc) initTurbulence(vgrid, 8 nG, 60 kpc, 800 kpc, -11. / 3, 42) Bfield = MagneticFieldGrid(vgrid)
simulation setup sim = ModuleList() sim.add(PropagationCK(Bfield)) sim.add(FutureRedshift()) sim.add(PhotoPionProduction(CMB(), photons, neutrinos, electrons)) sim.add(ElectronPairProduction(CMB())) sim.add(ElectronPairProduction(IRB_Finke10())) sim.add(EMPairProduction(URB_Protheroe96(),True)) sim.add(EMInverseComptonScattering(CMB(),True)) sim.add(EMDoublePairProduction(URB_Protheroe96(),True)) sim.add(EMInverseComptonScattering(URB_Protheroe96(),True)) sim.add(EMTripletPairProduction(URB_Protheroe96(),True)) sim.add(MinimumEnergy(1 * EeV))
periodic boundaries extent = 512 30 kpc # size of the magnetic field grid sim.add(PeriodicBox(Vector3d(-extent), Vector3d(2 * extent)))
define the observer obs = Observer() obs.add(ObserverSmallSphere(Vector3d(0, 0, 0), 0.5 * Mpc)) obs.add(ObserverRedshiftWindow(-0.05, 0.05)) output = TextOutput(output1.txt, Output.Event3D) output.disableAll() # disable everything to start from scratch output.enable(Output.CurrentEnergyColumn) # current energy output.enable(Output.CurrentIdColumn) # current particle type output.enable(Output.SourceEnergyColumn) # current energy output.enable(Output.SourceIdColumn) # current particle type
obs.onDetection(output) sim.add(obs)
define the source(s) source = Source() source.add(SourcePosition(Vector3d(50, 0, 0) Mpc)) source.add(SourceIsotropicEmission()) source.add(SourceParticleType(nucleusId(1, 1))) source.add(SourcePowerLawSpectrum(1 EeV, 10000 * EeV, -2.0)) source.add(SourceRedshiftEvolution(1.5, 0.001, 3))
run simulation sim.setShowProgress(True) sim.run(source, 5000000) `
sim2.py, sim3.py, etc are the same copies of sim1.py with different output file names. I am trying to run sim1.py, sim2.py etc on different nodes simultaneously.
Suppose, I have five CRPropa codes sim1.py ....sim5.py
I am using a SLURM script as below to run these five codes
`#!/bin/bash
for i in {1..5} do python sim$i.py done
` I want to send the codes sim1.py ........sim5.py to the 5 different nodes that I have chosen. But this way it doesn't work. Perhaps the issue is with the SLURM script.
I am actually new to cluster computing. So, if you can provide some information on how to do this, it would be really helpful!
The issue is in your slurm script, you are executing all scripts on all nodes. A minimum working example (although quite tedious) would be to create 5 slurm scripts that submit one job to one node. Please ask you IT Team respectively cluster administrator for an introduction into your system or look for online tutorials.
Thanks a lot! Just to clarify, are you talking about creating JOB array?
Hi,
I wish to run CRPropa on cluster using multiple nodes. CRPropa can use all the processors available in a single node as it has OpenMP support. But it can not utilise processors across different nodes (perhaps, it requires MPI support). Therefore, I want to distribute multiple copies of CRPropa simulation code in different nodes using a single JOB script. For example, I have 3 cluster nodes: node1, node2, node3 and I want to run 3 CRPropa codes (say Sim1.py, Sim2.py, Sim3.py) in these nodes simultaneously in such a way that Sim1.py goes to node1, Sim2.py goes to node2 etc.
The job submission is done using SLURM script. I tried a few SLURM scripts to run CRPropa in the way that I have described above, but it didn't work. Is it possible to run CRPropa in this way? Can you help me to figure out a SLURM script which can fulfil my purpose. Any alternate suggestions would also be very helpful.
Thank you!