Distributed algorithm segmentation fault (using SLURM + MVAPICH2)

sunrise2575 commented 6 years ago

Hi,

I'm using MVAPICH 2.3b and SLURM 17.11.4 (with MUNGE, MariaDB, OpenSSL, PAM, etc....) on CentOS 7.4

I use three nodes. (Intel Xeon CPU cluster) sun07 is the master, which has slurmctld and slurmd daemon process, and sun08, sun09 are slaves, which only have slurmd daemon process.

MVAPICH compiling & installing was conducted by ./configure --with-pmi=pmi2 --with-pm=slurm as mentioned on the MVAPICH2 manual. (http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3b-userguide.pdf)

I tested linux command hostname and _mpihello.c. (https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_hello.c)

[heeyong@sun07 ~] srun -n 3 hostname
srun: job88 queued and waiting for resources
srun: job88 has been allocated resources
sun07
sun08
sun09

[heeyong@sun07 ~] srun -n 3 ./mpi_hello
srun: job 89 queued and waiting for resources
srun: job 89 has been allocated resources
Hello from task 0 on sun07!
MASTER: Number of MPI tasks is: 3
Hello from task 2 on sun09!
Hello from task 1 on sun08!

Those two tests were perfectly worked in multi-node situation.

However, If I run command like this:

[heeyong@sun07 gemini]$ srun -n 3 ./toolkits/pagerank /path/to/twitter-2010.binedgelist 41652230 20
srun: job 95 queued and waiting for resources
srun: job 95 has been allocated resources
[sun09:mpi_rank_2][error_sighandler] Caught error: Segmentation fault (signal 11)
srun: error: sun09: task 2: Segmentation fault (core dumped)
srun: error: sun08: task 1: Segmentation fault (core dumped)

and no response anymore.

I'm sure that I clearly compiled and executed Gemini on the same MVAPICH version, like you mentioned at https://github.com/thu-pacman/GeminiGraph/issues/2

Finally, I traced manually (using printf) and I found that the program stucks at 1082~1102 line of core/graph.hpp

What is the problem...?

coolerzxw commented 6 years ago

Hi @sunrise2575 , is the input file properly encoded in the binary edge list format (i.e. each edge containing two integers in binary)? By the way, you can first try running the program using a single process (i.e. without "srun -n [N]") and see if it behaves normally.

sunrise2575 commented 6 years ago

Software

CentOS 7.4
GCC 6.4.0
MVAPICH 2.3b
SLURM 17.11.5

Hardware

Intel Xeon E5-2630 v4
Mellanox InfiniBand ConnectX-4 EDR
512GB Memory

I downloaded:

twitter-2010 graph (http://data.law.di.unimi.it/webdata/twitter-2010/twitter-2010.graph)
twitter-2010 properties (http://data.law.di.unimi.it/webdata/twitter-2010/twitter-2010.properties)

I decompressed both file to text edge list. using this code (https://github.com/btootoonchi/Anchored_2-Core)

The result was clean and sorted.

[heeyong@sun07 dataset]$ head twitter-2010.txt
0 99732
1 0
2 3
2 13733359
3 2
3 38698319
4 5
4 13679061
5 4
5 6

I converted text edge list to binary edge list using my program with reference to GraphChi: (https://github.com/GraphChi/graphchi-cpp/blob/master/src/preprocessing/conversions.hpp)

Here is the GraphChi code section.

template <typename EdgeDataType, typename FinalEdgeDataType>
void convert_binedgelist(std::string basefilename, sharder<EdgeDataType, FinalEdgeDataType> &sharderobj) {
    std::vector<std::string> parts;
    std::string dirname = get_dirname(basefilename);
    std::string prefix =  get_filename(basefilename);

    std::cout << "dir=[" << dirname << "] prefix=[" << prefix << "]" << std::endl;
    getdir(dirname, parts);

    for(std::vector<std::string>::iterator it=parts.begin(); it != parts.end(); ++it) {
        std::string inputfile = *it;
        if (inputfile.find(prefix) == 0 && inputfile.find("tmp") == inputfile.npos) {
            std::cout << "Going to process: " << inputfile << std::endl;
        }
    }

    for(std::vector<std::string>::iterator it=parts.begin(); it != parts.end(); ++it) {
        std::string inputfile = *it;
        if (inputfile.find(prefix) == 0 && inputfile.find(".tmp") == inputfile.npos) {
            inputfile = dirname + "/" + inputfile;
            std::cout << "Process: " << inputfile << std::endl;
            FILE * inf = fopen(inputfile.c_str(), "r");

           ////////////////// THIS PART //////////////////
            while(!feof(inf)) {
                vid_t from;
                vid_t to;

                size_t res1 = fread(&from, sizeof(vid_t), 1, inf);
                size_t res2 = fread(&to, sizeof(vid_t), 1, inf);

                assert(res1 > 0 && res2 > 0);
                if (from != to) {
                    sharderobj.preprocessing_add_edge(from, to, EdgeDataType());
                }
            }
            fclose(inf);
            //////////////////////////////////////////////////
        }
    }
}

Text numbers are converted to uint32_t by little endian byte order. Self-loop edges are removed.

Here is converted result.

[heeyong@sun07 dataset]$ xxd -l 160 twitter-2010.binedgelist
0000000: 0000 0000 9485 0100 0100 0000 0000 0000  ................
0000010: 0200 0000 0300 0000 0200 0000 ef8d d100  ................
0000020: 0300 0000 0200 0000 0300 0000 4f7d 4e02  ............O}N.
0000030: 0400 0000 0500 0000 0400 0000 d5b9 d000  ................
0000040: 0500 0000 0400 0000 0500 0000 0600 0000  ................
0000050: 0600 0000 ada6 0100 0700 0000 0800 0000  ................
0000060: 0700 0000 0900 0000 0700 0000 9bb6 0100  ................
0000070: 0800 0000 0700 0000 0900 0000 0700 0000  ................
0000080: 0a00 0000 0b00 0000 0b00 0000 0a00 0000  ................
0000090: 0b00 0000 0c00 0000 0b00 0000 36b8 0100  ............6...

Then, the 1 node test looks good. (I used 10 iteration instead of 20)

$ git clone https://github.com/thu-pacman/GeminiGraph
$ cd GeminiGraph
$ make
$ srun -n 1 ~/GeminiGraph/toolkits/pagerank ~/dataset/twitter-2010.binedgelist 41652230 10
srun: job 139 queued and waiting for resources
srun: job 139 has been allocated resources
Error in system call pthread_mutex_destroy: Device or resource busy
src/mpi/init/initthread.c:241
delta(0)=1.000000
delta(1)=1.286916
delta(2)=0.820889
delta(3)=0.441322
delta(4)=0.230969
delta(5)=0.120882
delta(6)=0.065829
delta(7)=0.035382
delta(8)=0.020318
delta(9)=0.011524
exec_time=5.173558(s)
pr_sum=38497680.114544
pr[21513299]=14083.346790
delta(0)=1.000000
delta(1)=1.286916
delta(2)=0.820889
delta(3)=0.441322
delta(4)=0.230969
delta(5)=0.120882
delta(6)=0.065829
delta(7)=0.035382
delta(8)=0.020318
delta(9)=0.011524
exec_time=4.988416(s)
pr_sum=38497680.114544
pr[21513299]=14083.346790
delta(0)=1.000000
delta(1)=1.286916
delta(2)=0.820889
delta(3)=0.441322
delta(4)=0.230969
delta(5)=0.120882
delta(6)=0.065829
delta(7)=0.035382
delta(8)=0.020318
delta(9)=0.011524
exec_time=4.944450(s)
pr_sum=38497680.114544
pr[21513299]=14083.346790
delta(0)=1.000000
delta(1)=1.286916
delta(2)=0.820889
delta(3)=0.441322
delta(4)=0.230969
delta(5)=0.120882
delta(6)=0.065829
delta(7)=0.035382
delta(8)=0.020318
delta(9)=0.011524
exec_time=4.987351(s)
pr_sum=38497680.114544
pr[21513299]=14083.346790
delta(0)=1.000000
delta(1)=1.286916
delta(2)=0.820889
delta(3)=0.441322
delta(4)=0.230969
delta(5)=0.120882
delta(6)=0.065829
delta(7)=0.035382
delta(8)=0.020318
delta(9)=0.011524
exec_time=4.955934(s)
pr_sum=38497680.114544
pr[21513299]=14083.346790
delta(0)=1.000000
delta(1)=1.286916
delta(2)=0.820889
delta(3)=0.441322
delta(4)=0.230969
delta(5)=0.120882
delta(6)=0.065829
delta(7)=0.035382
delta(8)=0.020318
delta(9)=0.011524
exec_time=4.978177(s)
pr_sum=38497680.114544
pr[21513299]=14083.346790

However, the 3 node test was failed by segmentation fault.


$ srun -N 3 ~/GeminiGraph/toolkits/pagerank ~/dataset/twitter-2010.binedgelist 41652230 10
srun: job 141 queued and waiting for resources
srun: job 141 has been allocated resources
[sun09:mpi_rank_2][error_sighandler] Caught error: Segmentation fault (signal 11)
[sun07:mpi_rank_0][error_sighandler] Caught error: Segmentation fault (signal 11)
srun: error: sun09: task 2: Segmentation fault (core dumped)
srun: error: sun07: task 0: Segmentation fault (core dumped)

^Csrun: interrupt (one more within 1 sec to abort) srun: step:141.0 task 1: running srun: step:141.0 tasks 0,2: exited abnormally ^Csrun: sending Ctrl-C to job 141.0 srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

Because the gemini gave no more responses during 10 minutes, I killed process manually.
By using top, I figured out that after showing segmentation fault,
it constantly uses exactly 2 cores with 100% utilization ... per each node.

**Conclusion**
- 1 node test is good
- 3 node test is bad
- the binary edge list doesn't have problem

**Plus**
I tried very small test using this text edge list:

0 1 1 2 2 3 3 4 4 0


which has five vertices and the overall topology is circular.
It works well in 1 node test, but not 3 node test... segmentation fault.

Please help me

coolerzxw commented 6 years ago

"By using top, I figured out that after showing segmentation fault, it constantly uses exactly 2 cores with 100% utilization ... per each node." According to the above behavior and your MPI environment being MVAPICH2, I think the problem might be that MPI_Init_thread(argc, argv, MPI_THREAD_MULTIPLE, &provided) does not return MPI_THREAD_MULTIPLE as required. You can try passing "MV2_ENABLE_AFFINITY=0" to run the command (i.e. before srun) according to this discussion.

sunrise2575 commented 6 years ago

MV2_ENABLE_AFFINITY=0 solves the whole problem. Thanks so much!

thu-pacman / GeminiGraph

Distributed algorithm segmentation fault (using SLURM + MVAPICH2) #5