Closed afranques closed 3 years ago
@afranques, could you tell me which branch you are using?
Hello @sknigh! I am using the master branch.
Thanks, Antonio
@sknigh, I tried allocating the array dynamically (using malloc
) instead of statically, and it now works! I still don't know why large statically-allocated arrays cause the segmentation fault in the simulator and not when run natively in the machine, but at least it works (for now) when the array is dynamically-allocated. I am attaching the new code here in case it helps anyone in the future ;-)
Thanks to anyone who took a look at my post!
#include <mpi.h>
#include <stdio.h>
// We have 1536 ranks (24 cores per node, 4x4x4 nodes)
#define NRANKS 1536
#define NELEMS_PER_RANK 20
#define NELEMS (NRANKS*NELEMS_PER_RANK)
#define ROOT 0
int main(int argc, char** argv) {
int rank, size;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
int nelems = NELEMS; // total number of elements (each of 4 bytes) we have in the send buffer
int* sendbuf;
int* recvbuf;
// Only the root needs a properly sized send buffer
if (rank == ROOT) {
sendbuf = (int*) malloc(NELEMS * sizeof(int));
}
// All other ranks simply set up a dummy send buffer with one element (which won't be used)
else {
sendbuf = (int*) malloc(1 * sizeof(int));
}
recvbuf = (int*) malloc(NELEMS_PER_RANK * sizeof(int));
// Check if the memory has been successfully allocated by malloc or not
if (sendbuf == NULL || recvbuf == NULL) {
printf("Rank %d: Memory not allocated.\n", rank);
exit(0);
}
// The root fills the send buffer with the values we want to scatter
if (rank == ROOT){
for (int i=0; i < nelems; ++i){
// If NELEMS_PER_RANK = 4 --> sendbuf = [0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3]
// This is done so that each rank will send a group of elements all with same value (so receiver can check after scatter)
sendbuf[i] = i/NELEMS_PER_RANK;
// printf("Root: sendbuf[%d]=%d\n", i, sendbuf[i]);
}
printf("Size of 1 element: %zu bytes\n", sizeof(sendbuf[0]));
printf("Number of elements sent per rank: %d (%zu bytes)\n", NELEMS_PER_RANK, NELEMS_PER_RANK*sizeof(sendbuf[0]));
printf("Size of sendbuf: %zu bytes (%d elements)\n", NELEMS*sizeof(sendbuf[0]), NELEMS);
}
// Call is same for root and others, so all ranks call the same function
// We scatter NELEMS_PER_RANK to each rank (of MPI_INT type each element) from ROOT's sendbuf to each rank's recvbuf
MPI_Scatter(sendbuf, NELEMS_PER_RANK, MPI_INT, recvbuf, NELEMS_PER_RANK, MPI_INT, ROOT, MPI_COMM_WORLD);
// All ranks check they received the data they were supposed to
for (int i=0; i < NELEMS_PER_RANK; i++) {
if (rank != recvbuf[i]) {
printf("Rank %d has NOT received the expected data (instead received recvbuf[%d]=%d)\n", rank, i, recvbuf[i]);
}
}
if (rank == ROOT){
printf("Rank %d finished at t = %8.4f ms\n", ROOT, MPI_Wtime()*1e3);
}
MPI_Finalize();
if (rank == ROOT){
printf("Rank %d passed MPI_Finalize\n", ROOT);
}
free(sendbuf);
free(recvbuf);
return 0;
}
Yes, I'm sure you were overrunning the stack size in the user space threads that macro uses to model application tasks. The stack size can be increased, but malloc is probably the better way to go. Sorry we weren't able to get to the issue sooner, people are spread pretty thin at the moment...
Ahh, I see, that makes sense. Thank you very much for your reply, @jpkenny! And please don't be sorry at all, we are lucky to have you even look at these things! I should have tried the malloc
earlier; bugs always seem so obvious in retrospect haha
Hello! I was trying to characterize the performance of
MPI_Scatter
over different message sizes using SST-Macro with the integrated SST-Core, and I realized that the simulator crashes when the sending buffer is larger than ~128 KB (~32,000MPI_INT
elements of 4 bytes each).The application I am running is the following (which I call
main.cc
), and it consists of a simpleMPI_Scatter
call with a few bells and whistles around it:This is the Makefile I use:
This is the parameters file I use (which I name
parameters.ini
):Which uses the
small_torus.ini
configuration that comes with the simulatorI have all these files in the same folder, so I just compile by typing:
$> make
And I run from inside this same folder with:$> pysstmac -f parameters.ini
The output I get is:
As mentioned at the beginning, this segmentation fault happens when
NELEMS_PER_RANK
is above20
, which assuming we have 1536 ranks in total, and each element is 4 bytes, it means it happens whensendbuf
is above around1536 x 20 x 4 ~= 128 KBytes
. WhenNELEMS_PER_RANK
is lower or equal than20
the simulation finishes successfully.Any help will be highly appreciated!
Thanks, Antonio