Closed adigitoleo closed 6 months ago
Example:
#!/bin/bash
#PBS -P xd2
#PBS -q normalbw
#PBS -l walltime=05:00:00
#PBS -l ncpus=168
#PBS -l mem=1536GB
#PBS -l jobfs=2400GB
#PBS -l storage=scratch/xd2
#PBS -l wd
#PBS -o simple3d_unsteady_prescr_dc1000000.log
#PBS -e simple3d_unsteady_prescr_dc1000000.err
#PBS -N simple3d_unsteady_prescr_dc1000000
# Install pydrex into this python environment.
module purge
module load python3/3.11.7 python3-as-python
ulimit -c unlimited
# Copy the tools/pbs_start_ray*.sh into $PWD.
source pbs_start_ray_cluster.sh
# Run this script from outside the PyDRex tree, which should be in `PyDRex/*`.
ray job submit --working-dir PyDRex/ -- \
python3 -m pytest tests/test_simple_shear_3d.py \
--outdir=$PBS_O_WORKDIR/_out_1 -v --runslow --ncpus=$PBS_NCPUS -k="direction_change[1]"
ray stop
multiprocessing.Pool
API that is very similar to the already supported Python stdlibmultiprocessing
stuffimap
inside the M-index calculation routines cuts down the memory even more, at the cost of extra CPU time (which can be gained back by running on more nodes now)ray.put
andray.get
calls, but doing the same for otherslow
tests would look the same