FXXXXXX666
commented
4 months ago Hello Edward!
I am glad to see your message. Upon reviewing the program, I have identified some issues.
Firstly, you did not use the original code from GitHub for testing, but rather used the code from Zenodo.
PanguLU is inherently a distributed heterogeneous direct solver. The code on Zenedo is primarily intended to replicate the performance of our experiments on a large-scale distributed computing machine.
If you have more GPUs, such as eight A100 40GB GPUs, the total computation time on each GPU will match the experimental performance in our paper when summed.
You pointed out our code on Zenodo, and earlier tests revealed that for some matrices, running on a small number of A100 40GB GPUs would cause out-of-memory issues. To avoid this, we added new checks in the pangulu_preprocess.h file compared to the original GitHub code.
This approach causes PanguLU to allocate a very small segment of GPU memory initially, placing the required computation space in CPU memory and transferring it to the GPU when needed. This leads to additional GPU-to-CPU bandwidth overhead, resulting in the situation you described. The code is as follows:
GPU_MEMORY_FLAG=0;
if(sum_rank_size==1){
int_t Big_GPU_Memory=((int_t)1000*1000*1000)*8;
if(CPU_MEMORY>(Big_GPU_Memory)){
GPU_MEMORY_FLAG=1;
}
}
else if(sum_rank_size<=2){
int_t Big_GPU_Memory=((int_t)1000*1000*1000)*4;
if(CPU_MEMORY>(Big_GPU_Memory)){
GPU_MEMORY_FLAG=1;
}
}
else if(sum_rank_size<=4){
int_t Big_GPU_Memory=((int_t)1000*1000*1000)*20;
if(CPU_MEMORY>(Big_GPU_Memory)&&(N>=500000)){
GPU_MEMORY_FLAG=1;
}
}If you use eight GPUs, this situation will not occur, so I recommend using eight GPUs if you have more A100 GPU cards available. This is also the easiest modification to make.
Additionally, a single A100 80GB GPU can also be used for testing. You only need to delete this segment of code and force GPU_MEMORY_FLAG to 0, as shown below, to ensure that the performance matches what is described in our paper:
GPU_MEMORY_FLAG=0;
/*
if(sum_rank_size==1){
int_t Big_GPU_Memory=((int_t)1000*1000*1000)*8;
if(CPU_MEMORY>(Big_GPU_Memory)){
GPU_MEMORY_FLAG=1;
}
}
else if(sum_rank_size<=2){
int_t Big_GPU_Memory=((int_t)1000*1000*1000)*4;
if(CPU_MEMORY>(Big_GPU_Memory)){
GPU_MEMORY_FLAG=1;
}
}
else if(sum_rank_size<=4){
int_t Big_GPU_Memory=((int_t)1000*1000*1000)*20;
if(CPU_MEMORY>(Big_GPU_Memory)&&(N>=500000)){
GPU_MEMORY_FLAG=1;
}
}
*/I also noticed some issues with your testing platform. You are using an Intel Xeon E5-2695, which has the following specifications:
Processor Number: E5-2695 v2
Number of Cores: 12
Number of Threads: 24
Base Frequency: 2.4 GHzIn our experiments, we used the Kunpeng 920 7265, with specifications as follows:
Processor Number: Kunpeng 920 7265
Number of Cores: 32
Number of Threads: 32
Base Frequency: 2.6 GHzAdditionally, the different machines result in different PCIe bandwidths.
The performance discrepancy in your experiments might be due to differences in CPU performance and PCIe data transfer bandwidth. You can replicate our experiments testing each sparse BLAS performance.
Theoretically, our sparse BLAS should not have issues, and the problem likely lies with the CPU-to-A100 GPU bandwidth.
PanguLU is designed for distributed computing systems. To simplify the design, even a single GPU requires data to be copied to the CPU after computation, which can cause severe bandwidth issues with only one GPU.
edwardnjust
The GPU i used is NVIDIA A100 40GB, and CPU is Intel Xeon E5-2695 v4. I test the program on single GPU. These matrices are significantly slower than the paper : ASIC_680k(3.62x slower), cage12(10.18x slower), dielFilterV3real(3.38x slower), nlpkkt80(2.01x slower). Here is a result about cage12 matrix.
CUDA Runtime: 12030 CUDA Driver: 12000 MPI Processes 1 Matrix is ../matrix/cage12.tsv ADAPTIVE_KERNEL_SELECTION -------------ON SYNCHRONIZE_FREE ----------------------ON N is 130228 ,NNZ is 2032536 the gpu num is 2 the rank is 0 use the 0 gpu Device NVIDIA A100-PCIE-40GB Symbolic L + U NNZ = 570274806 the symbolic time is 1.003155 s ../matrix/cage12_00200/ read ../matrix/cage12_00200/ rank 0 the preprocess time is 127.743160 s rank 0 the numerical time is 578.705797 s 7.308713 GFLOPs filename = ../matrix/cage12.tsv matrix_name : cage12 rank getrf tstrf gessm ssssm cal_sum 0 0.40429 10.30226 7.36425 541.90883 559.97963 pangulu_test---------------------------finish
Why are my test results so much slower than in the paper?