276 combine nsf loops

[connoraird]

Description

• We have combined three consecutive loops over support functions (nsup) into a single one.
• This loop has then been threaded and collapsed
  • To allow this, some memory allocation had to be refactored and moved outside of the parallel region.

Speedup plot

This plot shows the performance of test test_004_isol_C2H4_4proc_PBE0CRI for 1 mpi process

[connoraird]

Results of running benchmarks CONQUEST 276 parallel speedup.pdf

To generate this data, from within a directory containing these files

• Conquest - Compiled executable

• submit-kathleen.sh - Script to submit jobs to kathleen (UCL cluster)

  #!/bin/bash -l
  
  #$ -l h_rt=1:00:00
  #$ -N conquest_vtune_hotspot
  #$ -pe mpi 41
  #$ -l mem=3G
  #$ -cwd
  
  APP=../../Conquest
  export OMP_NUM_THREADS=1
  export OMP_STACKSIZE=100M
  export MKL_NUM_THREADS=1i
  export LD_LIBRARY_PATH=/shared/ucl/apps/libxc/6.2.2/intel-2022/lib:$LD_LIBRARY_PATH
  
  mpiexec -np 1 $APP

• Each of the test_* benchmark directories

I ran the following script.

for d in test_*/ ; do
    echo "$d"
    cd $d
    for i in 1 2 4 8 16; do
        echo "np-1-nt-$i"
        mkdir np-1-nt-$i
        cp C* H* ../submit-kathleen.sh np-1-nt-$i
        cd np-1-nt-$i
        sed -i "s/export OMP_NUM_THREADS=.*/export OMP_NUM_THREADS=${i}/g" "submit-kathleen.sh"
        qsub submit-kathleen.sh
        cd ../
    done
    cd ../
done

Once each job had completed, I ran the following script to gather the data into a format I could work with.

for d in */ ; do
        for i in 1 2 4 8 16; do
                echo "$d, $i, " | tr '\n' ' ' >> data.csv
                cat $d/np-1-nt-$i/Conquest_out | grep "Total run time" | grep -Eo '[+-]?[0-9]+([.][0-9]+)' | tr '\n' ' ' >> data.csv
                echo ', ' | tr '\n' ' ' >> data.csv
                cat $d/np-1-nt-$i/time.001 | grep "Exact" | grep -Eo '[+-]?[0-9]+([.][0-9]+)' | tr '\n' ' ' >> data.csv
                echo ', ' | tr '\n' ' ' >> data.csv
                cat $d/np-1-nt-$i/exx_timers.001 | grep "nsup" | grep -Eo '[+-]?[0-9]+([.][0-9]+)' >> data.csv
        done
        echo ' ' >> data.csv
done

[lionelalexandre]

Dear Connor, results are impressive, that's a major achievement ! Thank you.