wrong endianness for "external32" representation

[tgastine]

It seems that the endianness is always shifted to "native" in Open MPI MPI-IO implementation for the external32 data representation. Here is a little fortran program compiled using mpif90 -fconvert=big-endian test.f90

program test

   use mpi

   implicit none

   integer :: rank, n_procs
   integer :: ierr, fh
   integer :: istat(MPI_STATUS_SIZE)
   integer(KIND=MPI_OFFSET_KIND) :: disp

   integer :: l_max
   real(kind=4) :: val

   l_max=4
   val = 2.0

   call MPI_init(ierr)

   call MPI_comm_rank(MPI_COMM_WORLD,rank,ierr)
   call MPI_comm_size(MPI_COMM_WORLD,n_procs, ierr)

   call MPI_File_Open(MPI_COMM_WORLD, 'test', ior(MPI_MODE_WRONLY,MPI_MODE_CREATE), &
        &             MPI_INFO_NULL, fh, ierr)

   if ( rank == 0 ) then
      disp = 0
   else
      disp = 8
   end if
   call MPI_FILE_SET_VIEW(fh, disp, MPI_BYTE, MPI_BYTE,"external32", &
       &                  MPI_INFO_NULL, ierr) 

   if ( rank == 0 ) then
      call MPI_FILE_WRITE(fh, l_max, 1, MPI_INTEGER, istat, ierr)
      call MPI_FILE_WRITE(fh, val, 1, MPI_REAL, istat, ierr)
      call MPI_FILE_WRITE(fh, l_max, 1, MPI_INTEGER, istat, ierr)
   end if

   call MPI_File_close(fh, ierr)

   call MPI_Finalize(ierr)

end program test

When I try to read it I got a little-endian binary file. This works fine with mpich2 3.1 and with intelmpi. I know it used to be an issue in the old OpenMPI version (< 1.8), but on my latest test on the 2.0 branch it still behaves the same. Is there any plan to have the external32 representation supported at some point?

Best regards,

Thomas

[edgargabriel]

The support for different data representations in the OMPIO implementation is incomplete to be honest. There are bits and pieces that are done, but also significant portions of the code that are missing to support this functionality. It is simply a question of manpower to get that part done. Do we have plans to support that feature? Yes. But I will have a hard time to give a firm date when the functionality is available.

The only recommendation that I have at this point is that you can try to switch to the romio314 module and see whether it works with that, i.e.

mpirun --mca io romio314 -np ...

Thanks

Edgar

On 2/15/2017 8:28 AM, Thomas Gastine wrote:

It seems that the endianness is always shifted to "native" in Open MPI MPI-IO implementation for the |external32| data representation. Here is a little fortran program compiled using |mpif90 -fconvert=big-endian test.f90|

program test

use  mpi

implicit none

integer  ::  rank, n_procs
integer  ::  ierr, fh
integer  ::  istat(MPI_STATUS_SIZE)
integer(KIND=MPI_OFFSET_KIND)::  disp

integer  ::  l_max
real(kind=4)::  val

l_max=4
val=  2.0

call  MPI_init(ierr)

call  MPI_comm_rank(MPI_COMM_WORLD,rank,ierr)
call  MPI_comm_size(MPI_COMM_WORLD,n_procs, ierr)

call  MPI_File_Open(MPI_COMM_WORLD,'test',ior(MPI_MODE_WRONLY,MPI_MODE_CREATE), &
     &             MPI_INFO_NULL, fh, ierr)

if  ( rank ==0  )then
   disp=  0
else
   disp=  8
end if
call  MPI_FILE_SET_VIEW(fh, disp, MPI_BYTE, MPI_BYTE,"external32", &
    &                  MPI_INFO_NULL, ierr)

if  ( rank ==0  )then
   call  MPI_FILE_WRITE(fh, l_max,1, MPI_INTEGER, istat, ierr)
   call  MPI_FILE_WRITE(fh, val,1, MPI_REAL, istat, ierr)
   call  MPI_FILE_WRITE(fh, l_max,1, MPI_INTEGER, istat, ierr)
end if

call  MPI_File_close(fh, ierr)

call  MPI_Finalize(ierr)

end program test

When I try to read it I got a little-endian binary file. This works fine with mpich2 3.1 and with intelmpi. I know it used to be an issue in the old OpenMPI version (< 1.8), but on my latest test on the 2.0 branch it still behaves the same. Is there any plan to have the |external32| representation supported at some point?

Best regards,

Thomas

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/open-mpi/ompi/issues/2985, or mute the thread https://github.com/notifications/unsubscribe-auth/AH22Hh3SSy-XsOTxdyPnBD564gAxPoV4ks5rcwuAgaJpZM4MBx14.

[tgastine]

Thanks for this feedback. Unfortunately the romio314 module doesn't change anything.

[ggouaillardet]

@tgastine some backport was missing from master i filled #2996 and #2997 for that. note only romio314 correctly supports external32

[tgastine]

Thanks for that. That's great to hear.