ornladios / ADIOS2-Examples

Application examples for the ADIOS2 I/O library https://github.com/ornladios/ADIOS2. This is Work in Progress.
Apache License 2.0
20 stars 16 forks source link

Read -1, expected 5285, errno = 1 #62

Closed xiaoruiDu closed 2 years ago

xiaoruiDu commented 2 years ago

Dear ADIOS2 Community, I am runing gray-scott example. The program can run without crash but give me this read error when I launch it ` mpirun --allow-run-as-root -n 4 adios2-gray-scott settings-files.json

[de44f6097ee6:39278] Read -1, expected 5285, errno = 1 [de44f6097ee6:39279] Read -1, expected 5285, errno = 1 [de44f6097ee6:39280] Read -1, expected 5285, errno = 1

` any idea?

more output:

31F73B13-DB60-45E5-A0D7-97ED2DA2D981
pnorbert commented 2 years ago

What version/commit of ADIOS2 and ADIOS2-Examples are you using? Why do you need that root flag for mpirun? Are you sure your are running the executable file that you expect, or may the system find another version in your path?

I have never seen this error, and I just tried with ADIOS2 and ADIOS2-Examples master branch but I don't get anything like this.

xiaoruiDu commented 2 years ago

hey, thanks for you fast reply. I run adios2(adios2-example) in docker container '--allow-run-as-root' is for docker reason.

I also use adios2 master branch. Is it possible that this error is only for docker?

xiaoruiDu commented 2 years ago

by the way, it can run without any error if i set process number to 1.
mpirun --allow-run-as-root -n 1 adios2-gray-scott settings-files.json.

pnorbert commented 2 years ago

Do you get this error messages if you run other programs with multiple processes? The error message does not come from ADIOS2.

pnorbert commented 2 years ago

see https://github.com/horovod/horovod/issues/503

xiaoruiDu commented 2 years ago

see horovod/horovod#503

thanks for this hint. I already spent 2 days on this error. Now, problem solved with setting enviroment varible before lunching the job.

export OMPI_MCA_btl_vader_single_copy_mechanism=none

https://github.com/open-mpi/ompi/issues/4948#issuecomment-395468231