xuanweishan
commented
1 year ago Login node installation. (CentOS 7)
Install system in raid 1 ref. https://www.thegeekdiary.com/how-to-install-centos-rhel-7-on-raid-1-partition/
- Installation system
- Installation destination
- Choose both disks we want to build raid.
- Choose
I will configure partitioning- Click
create them automatically.- Choose
/homeand click-to delete it.- Select
Device Typeto beRAID, chooseRAID LEVELtoRAID1and write your personal label, such asmd0.- Similarly, you can set the Device type and labels for
swapand/bootpartitions- Installation summary:
- [x] Additional Development
- [x] Compatibility Libraries
- [x] Development Tools
- [x] Email Server
- [x] Emacs
- [x] File and Storage Server
- [x] Hardware Monitoring Utilities
- [x] Identity Management Server
- [x] Infiniband Support
- [x] KDE
- [x] Legacy X Window System Compatibility
- [x] Network File System Client
- [x] Platform Development
- [x] Python
- [x] Technical Writing
- [x] Security Tools
- [x] System Administration Tools
- Set up network
Edit/etc/sysconfig/network-scripts/ifcfg-XXX, where XXX is usually "enpXXX"
BOOTPROTO=static ONBOOT=yes IPADDR=192.168.0.200 GATEWAY=192.168.0.1 NETMASK=255.255.255.0 DNS1=140.112.254.4- NFS client
- Mount all remote folders
Do not mount/softwaremount /software; mount /home; mount /work1; mount /project; mount /projectX; mount /projectY; mount /projectZ- Install the CUDA driver
- Copy initialization scripts
cp /work1/shared/spock/init_script/*.sh /etc/profile.d/ cp /work1/shared/spock/init_script/*.csh /etc/profile.d/
cp /work1/shared/spock/etc/rc.local /etc/rc.d/
chmod +x /etc/rc.d/rc.local
```
Edit `/etc/rc.d/rc.local` as follows
1. Comment out the line `/usr/bin/nvidia-persistenced --verbose`
2. Comment out the line `nvidia-cuda-mps-control -d`
3. Replace `nvidia-smi -i 0 -c EXCLUSIVE_PROCESS` by `nvidia-smi -i 0 -c PROHIBITED`-
TORQUE
cd /work1/shared/spock/package/torque/src/torque-3.0.6Login node only: edit
Fish_Install.shto enable—enable-server- WARNING: do NOT run
Fish_Install.shin parallel (i.e., install one node at a time)sh Fish_Install.sh >& log.spockXX cd ../../etc cp pbs /etc/init.d/Edit
pbs.confto setstart_server=1andstart_mom=0cp pbs.conf /etc/ cp nodes_spock /var/spool/TORQUE/server_priv/nodes systemctl enable pbs source /etc/profile.d/torque.sh cd ../src/torque-3.0.6/ ./torque.setup root killall pbs_server systemctl start pbs # This error message is fine: "LOG_ERROR::No such file or directory (2) in read_config, fstat: config" systemctl status pbs
- WARNING: do NOT run
-
[Optional] Create the SSH key of root
ssh-keygen -t rsa cd ~/.ssh cp id_rsa.pub authorized_keys cp id_rsa* authorized_keys /work1/shared/spock/ssh_root/ -
NFS server
systemctl enable nfs systemctl start nfs cp /work1/shared/spock/etc/exports /etc/ exportfs -ra showmount -e spock00 # /software 192.168.0.0/24Comment out
spock00:/softwarein/etc/fstab -
CUDA
-
Driver
It should have been installed when following [[System Installation: Computing Node | System Installation: Computing Node]]
-
Libraries and samples
mkdir /software/cuda /software/cuda/12.1 ln -s /software/cuda/12.1 /software/cuda/defaultInstall 12.1
cd /work1/shared/spock/package/cuda/ sh cuda_12.1.0_530.30.12_linux.run --silent --toolkit --installpath=/software/cuda/12.1WIP -
[optional] cuDNN
- Ref: https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#install-linux
[Optional] Download the latest version
https://developer.nvidia.com/cudnn --> Download cuDNN- Installation (the following example adopts CUDA 10.1)
CUDNN_TMP=/work1/fish/Code_Backup/GPU_Cluster_Setup/eureka/package/cudnn/cudnn-10.1-linux-x64-v7.6.5.32 CUDA_TMP=/software/cuda/10.1 cp ${CUDNN_TMP}/include/cudnn.h ${CUDA_TMP}/include cp ${CUDNN_TMP}/lib64/libcudnn* ${CUDA_TMP}/lib64 chmod a+r ${CUDA_TMP}/include/cudnn.h ${CUDA_TMP}/lib64/libcudnn* - Test
Log in to a computing node first
Do NOT use GNU compiler later than 8
cd /tmp export CUDA_PATH=/software/cuda/10.1 export PATH=$CUDA_PATH/bin:$PATH export LD_LIBRARY_PATH=$CUDA_PATH/lib64:$LD_LIBRARY_PATH cp -r /work1/fish/Code_Backup/GPU_Cluster_Setup/eureka/package/cudnn/cudnn_samples_v7 . cd cudnn_samples_v7/mnistCUDNN make clean && make -j 16 ./mnistCUDNN # Test passed! cd ../../ rm -rf cudnn_samples_v7
- Installation (the following example adopts CUDA 10.1)
- Ref: https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#install-linux
-
Test
cd /software/cuda/12.0/NVIDIA_CUDA-10.2_Samples ./1_Utilities/deviceQuery/deviceQueryVerify that
- 1st line: "Detected 1 CUDA Capable device(s)"
- 2nd line: "Device 0: "GeForce RTX 3080 Ti"
- Last line: "Result = PASS"
./1_Utilities/bandwidthTest/bandwidthTestVerify that
- Host <-> Devince bandwidth ~13 GB/s.
- "Result = PASS"
-
[Optional] Reset the default mode to
graphical.target(a.k.a. runlevel 5)systemctl set-default graphical.target systemctl get-default # graphical.target
-
-
Intel compiler
-
Create directory and link for intel compiler.
mkdir /software/intel ln -s /software/intel /opt1.oneAPI Basic & HPC toolkit
-
Install Basic toolkit
cd /software/intel cp /work1/shared/spock/package/intel/l_BaseKit_p_2023.0.0.2553.sh ./ sudo bash l_BaseKit_p_2023.0.0.25537.sh -
Install HPC toolkit
cp /work1/shared/spock/package/intell_HPCKit_p_2023.0.0.25400.sh ./ sudo bash l_HPCKit_p_2023.0.0.25400.sh
-
-
After installation
mv /opt/intel/licenses /opt/intel/.pset /software/intel rm -rf /opt/intel cd /opt ln -s /software/intel cd /software/intel ln -s /work1/shared/spock/package/intel srcCheck
/etc/profile.d/intel.shand, if necessary, replaceoptbysoftware
-
-
Valgrind
mkdir -p /software/valgrind cd /work1/shared/spock/package/valgrind/valgrind-3.15.0 # [Optional] Edit Fish_Install.sh sh Fish_Install.sh >& log.spockAfter installation
cd /software/valgrind ln -s /work1/shared/spock/package/valgrind src ln -s 3.15.0 defaultCheck
/etc/profile.d/valgrind.sh -
OpenMPI
mkdir -p /software/openmpi cd /software/openmpi/src/openmpi-4.1.1 # [Optional] Edit Fish_Install_with_UCX.sh (remember to un-comment the configuration flags) sh Fish_Install_with_UCX.sh >& log.spock-intelAfter installation
- check linking to
UCXlibrariescd /software/openmpi/4.1.1-intel-oneapi/bin objdump -p mpicxx | grep PATH # see whether /software/openucx/ucx-1.12.0/lib is in RPATH ldd mpicxx | grep ucx # see whether dynamic linker can find UCX libraries - make soft links
cd /software/openmpi ln -s /work1/shared/spock/package/openmpi src unlink default # optional, if default already existed ln -s /software/openmpi/4.1.1-intel-oneapi default - Check
/etc/profile.d/openmpi.sh - Check debugger
ompi_info | grep "MCA memchecker" # MCA memchecker: valgrind (MCA v2.1.0, API v2.0.0, Component v3.1.5) - Set the
MCAparameters [Optional]- Edit the configure file
/software/openmpi/4.1.1-intel-oneapi/etc/openmpi-mca-params.confby root (for querying the path for configure file, one can use:ompi_info --params mca all --level 9 | grep mca_param_files). Add the below lines (2021/07/24):pml=ucx # include only ucx for pml osc=ucx # include only ucx for osc btl=^openib # exclude openib from btlwhich works for
OpenMPI 4.1.1andUCX 1.12.0, without giving warning message:[eureka01:45901] common_ucx.c:364 Warning: UCX is unable to handle VM_UNMAP event. This may cause performance degradation or data corruption. Pls try adding --mca opal_common_ucx_opal_mem_hooks 1 to mpirun/oshrun command line to resolve this issue. - Reference for excluding
openibifUCXlibrary is installed. - Reference for setting
MCAparameters by configure file (10. How do I set the value of MCA parameters?, 4. Files)
- Edit the configure file
- check linking to
-
Maui
cd /work1/fish/Code_Backup/GPU_Cluster_Setup/spock/package/maui/maui-3.3.1 # [Optional] Edit Fish_Install.sh sh Fish_Install.sh >& log.spock cd etc/ cp maui.d /etc/init.d/ cp maui.sh maui.csh /etc/profile.d/ systemctl enable maui.d cd /usr/local/mauiEdit
maui.cfgas follows (an example is put atmaui-3.3.1/maui.cfg.eureka)RMPOLLINTERVAL 00:00:15 #BACKFILLPOLICY FIRSTFIT #RESERVATIONPOLICY CURRENTHIGHEST #NODEALLOCATIONPOLICY MINRESOURCE # <==== Add by Nelson ====> JOBAGGREGATIONTIME 00:00:04 # Backfill BACKFILLPOLICY FIRSTFIT RESERVATIONPOLICY NEVER # Node Allocation NODEALLOCATIONPOLICY FIRSTAVAILABLE # Set Job Flags JOBACTIONONNODEFAILURE CANCEL JOBNODEMATCHPOLICY EXACTNODE systemctl start maui.d source /etc/profile.d/maui.sh -
Other packages
-
screen,pdshyum -y install screen yum -y install pdsh -
FFTWmkdir -p /software/fftw cd /work1/shared/spock/package/fftw/fftw-2.1.5-revised # [Optional] Edit Fish_Install.sh sh Fish_Install.sh >& log.spock-intelAfter installation
cd /software/fftw ln -s /work1/shared/spock/package/fftw src ln -s 2.1.5-intel default -
HDF5mkdir -p /software/hdf5 cd /work1/shared/spock/package/hdf5/hdf5-1.10.6 # [Optional] Edit Fish_Install.sh sh Fish_Install.sh >& log.spockAfter installation
cd /software/hdf5 ln -s /work1/shared/spock/package/hdf5 src ln -s 1.10.6 default -
GSLmkdir -p /software/gsl cd /work1/shared/spock/package/gsl/gsl-2.6 # [Optional] Edit Fish_Install.sh sh Fish_Install.sh >& log.spockAfter installation
cd /software/gsl ln -s /work1/shared/spock/package/gsl src ln -s 2.6 default -
gnuplotmkdir -p /software/gnuplot cd /work1/shared/spock/package/gnuplot/gnuplot-5.2.8 # [Optional] Edit Fish_Install.sh sh Fish_Install.sh >& log.spockAfter installation
cd /software/gnuplot ln -s /work1/shared/spock/package/gnuplot src ln -s 5.2.8 default -
Latest GNU compiler
- [Optional] Download the latest version
GCC_VERSION=9.3.0
wget https://ftp.gnu.org/gnu/gcc/gcc-${GCC_VERSION}/gcc-${GCC_VERSION}.tar.gz- Installation
mkdir -p /software/gcc cd /work1/shared/spock/package/gcc/gcc-9.3.0 # [Optional] Edit Fish_Install.sh sh Fish_Install.sh >& log.spockRef: https://gcc.gnu.org/install/configure.html
https://gcc.gnu.org/install/
https://gist.github.com/nchaigne/ad06bc867f911a3c0d32939f1e930a11 After installationcd /software/gcc ln -s /work1/shared/spock/package/gcc src ln -s 9.3.0 default
- Installation
- [Optional] Download the latest version
-
UCXlibrary- [Optional] Download the latest version
mkdir -p /software/openucx cd /software/openucx/ git clone https://github.com/openucx/ucx.git ucx- Installation
cd /software/openucx/ucx ./autogen.sh mkdir build cd build ../contrib/configure-release --prefix=/install_path_for_ucx --enable-mt #enable MPI_THREAD_MULTIPLE make && make installRef: https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX#%20UCX%20installation
Ref: https://github.com/openucx/ucx/issues/5284 EnableMPI_THREAD_MULTIPLEallows using parapllelytwithUCX, without the need to excludeucxfrompmlin the script.
- Installation
- [Optional] Download the latest version
-
-
Miscellaneous
-
Language
Add the following lines to/etc/environmentLANG=en_US.utf-8 LC_ALL=en_US.utf-8 -
Check
- After installation finished check it in the
applications>utilities>disksIt will show two disks with the same partition settings and 4 RAID-1 array under.
- After installation finished check it in the
-
Installation
Installation of hard wares
CPU: AMD Threadripper Pro 5975wxMB: ASUS Por WS WRX80 SAGE SE WIFI (Temporary)RAM: G.skill F4-3200C16-32GVK * 8GPU: MSI RTX 3080 Ti OCSSD: 500GPSU: 1200Wx. Install CUDA driver
sh /work1/xuanshan/cuda_11.7.1_515.65.01_linux.run --silent --driverX. InfiniBand driver./mlnxofedinstall --forceTemperature performance
CPU stress test
GPU stress test
work1/xuanshan/gpu-burn/gpu_burn