Open benjha opened 3 years ago
This is very impressive speed. Does it available on Andes? It seems even pigz is not available on Andes.
I've installed mpifileutils on Andes. To use it with SLURM add the next lines to your script:
export LD_LIBRARY_PATH=$WORLDWORK/stf218/benjha/andes/gcc_9.3.0/lib:$WORLDWORK/stf218/benjha/andes/gcc_9.3.0/lib64:$LD_LIBRARY_PATH
export PATH=$WORLDWORK/stf218/benjha/andes/gcc_9.3.0/bin:$PATH
The build scripts are in
/gpfs/alpine/world-shared/stf218/benjha/andes/code/mpifileutils
First execute dependencies.sh
and then mpifileutils.sh
, after setting CC_VER
, CLUSTER
, INSTALL_DIR_PREFIX
on each script.
The Andes version works great! But I encountered library issue with the Summit version. Probably due to the OS update. First, I have to load a slightly updated bzip2. Second, here is the libraries still missing:
bash-4.4$ ldd /gpfs/alpine/world-shared/stf218/benjha/summit/gcc_9.3.0/bin/dsync
/gpfs/alpine/world-shared/stf218/benjha/summit/gcc_9.3.0/bin/dsync: /lib64/liblzma.so.5: version `XZ_5.2.2' not found (required by /gpfs/alpine/world-shared/stf218/benjha/summit/gcc_9.3.0/lib/libarchive.so.13)
/gpfs/alpine/world-shared/stf218/benjha/summit/gcc_9.3.0/bin/dsync: /lib64/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by /gpfs/alpine/world-shared/stf218/benjha/summit/gcc_9.3.0/lib/libarchive.so.13)
yeah, Summit version is pre-Redhat 8.
I'll recompile it.
mpifileutils v0.10.1 and v0.11 is supported on Summit:, e.g.
module load gcc/9.1.0
module load mpifileutils/0.11
Datasets
bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt.tar.gz
272 GBafold_reduced_db_210726.tar.gz
147GBUncompressed
afold_reduced_db_210726.tar.gz
412GBTools
Base line numbers
Copying the 272 GB bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt.tar.gz file to NVMe in a job of one node took about 117 seconds.
For 32 nodes took about 106s and 114 s -> we are moving 32*272 GB = 8.7 TB in about 114s.
Decompression of the same file didn't finished in a lapse of 2 hrs.
Using mpifileutils
mpifileutils implements parallel sync which allows fast copies of complex directory trees (lots of directories with lots files of different sizes) as in
afold_reduced_db_210726.tar.gz
LSF Script pre-RedHat 8:
Sample output: