thanh-lam
commented
3 years ago Hello @jladd-mlnx , could you take a look? Thanks!
Open thanh-lam opened 3 years ago
thanh-lam
commented
3 years ago Hello @jladd-mlnx , could you take a look? Thanks!
jladd-mlnx
commented
3 years ago @thanh-lam - hi, we will take a look at this. @janjust - please take a look.
janjust
commented
2 years ago @thanh-lam Is this still an issue? I tried reproducing with the following, but on an x86 system [1]
FWIW I tried v4.0.x, v4.1.x , and v5.x but no segfault.
$oshrun --version
oshrun (OpenRTE) 4.0.7rc1
Report bugs to http://www.open-mpi.org/community/help/
tomislavj@helios010:/global/scratch/users/tomislavj/oshmem/tests
$ucx_info -v
# UCT version=1.12.1 revision dc92435
# configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --enable-mt --prefix=/global/scratch/users/tomislavj/oshmem/build-ucx/install --without-java
tomislavj@helios010:/global/scratch/users/tomislavj/oshmem/tests
$cat shmem_barrier.c
#include <stdio.h>
#include <stdlib.h>
#include <shmem.h>
#define BUF_LEN (1 * 1024 * 1024)
long pSync[SHMEM_ALLTOALL_SYNC_SIZE];
int main(void)
{
int me, pe_size, elms_per_task;
uint32_t *dst, *src;
size_t i;
shmem_init();
pe_size = num_pes();
me = my_pe();
dst = shmem_malloc(BUF_LEN);
src = shmem_malloc(BUF_LEN);
elms_per_task = BUF_LEN / sizeof(int) / pe_size;
for(i=0; i<SHMEM_ALLTOALL_SYNC_SIZE; i++ ) {
pSync[i] = SHMEM_SYNC_VALUE;
}
shmem_barrier_all();
if ( me < pe_size ) {
shmem_alltoall32( (void*)dst, (void*)src,
elms_per_task, 0, 0, pe_size, pSync );
}
shmem_barrier_all();
return 0;
}Hello @janjust ,
I just checked the version of oshrun on my systems and it is:
[/usr/mpi/gcc/openmpi-4.0.3rc4/bin]> ./oshrun --version
oshrun (OpenRTE) 4.0.3rc4
Report bugs to http://www.open-mpi.org/community/help/Can I try to verify with this level or should wait for 4.0.7rc1?
Thanks!
janjust
commented
2 years ago Can you try with v4.0.7? I'll try with 4.0.3.rc3, see if it'll reproduce for me
Thank you for taking the time to submit an issue!
Background information
What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
Open MPI was installed as part of the Mellanox software distribution.
If you are building/installing from a git clone, please copy-n-paste the output from
git submodule status.Please describe the system on which you are running
Details of the problem
This is a simple test program that calls shmem_alltoall32() for data communication. The test can be varied with different number of tasks and nodes.
Program source is available if needed. It's essentially does the following:
The oshrun command is as following: (To run all 32 tasks on one node)
That completed with no issue. But, if I added a second node and ran 31 tasks on the first and 1 task on the second, the test segfaulted.
That produced the following gdb trace stack on the second node: