AMPIF issues due to C++ main routine

[stwhite91]

Original issue: https://charm.cs.illinois.edu/redmine/issues/1268

---

When we use tlsglobals/static linking, the Fortran routine 'write' fails (though 'print' works). This is seemingly related to the fortran runtime not being properly initialized because Charm++'s main routine is C++, rather than Fortran. This issue has manifested differently before, such as Fortran-specific compiler flags being ignored since our main is in C++. For PlasComCM, the compilers set a flag to make all I/O be big endian, but AMPI ignores it, so we must set an environment variable at runtime to get big endian I/O...

[stwhite91]

Original date: 2017-04-12 04:54:20

---

This is actually going to be important for any AMPI Fortran code that we want to use tlsglobals with.

[PhilMiller]

Original date: 2017-04-14 22:31:10

---

I think we can provide a Fortran PROGRAM entry point that we'd compile as a stand-alone object file, that just calls whatever C/C++-based entry point, and link that in place of the C or C++-based main() that we use for C/C++ code when the linking step is launched as ampif90 or whatever

Does that sound like it would work?

[stwhite91]

Original date: 2017-08-30 23:05:54

---

^ That should work.

[stwhite91]

Original date: 2018-04-25 05:00:02

---

This may come up soon in trying to get -tlsglobals to work in all cases with Fortran via OpenMP's 'threadprivate' attribute.

[evan-charmworks]

Original date: 2018-05-17 18:39:30

---

I took a look at this and I'm not sure how to go about replacing the main in ck-core/main.C with a Fortran PROGRAM depending solely on ampif90 et al being the linker invocation. Instead, my intuition would have been something like the following:

1. Test at configure time if Fortran is available.
2. If so, always compile a Fortran PROGRAM into Charm++.
3. If not, continue using int main(int argc, char **argv).

[stwhite91]

Original date: 2018-05-17 19:11:22

---

That could also work. I think Phil's idea above was to compile Charm++/AMPI as a library without a main(), plus two separate object files with C++ main() and Fortran PROGRAM, then when linking the AMPI application, choose which main/PROGRAM to link in based on whether the linking is done with ampif90 or ampicc/ampicxx

[evan-charmworks]

Original date: 2018-05-17 19:50:20

---

One potential issue is that support for reading argv/argc from Fortran was only added in Fortran 2003, and converting the arguments from the interface Fortran provides to C-style argc/argv is beyond my understanding of Fortran.

I also need to test that running the program from a Fortran entry point will still allow C++ constructors of global variables to execute.

[stwhite91]

Original date: 2018-05-17 20:16:36

---

Yeah, I wouldn't be too surprised if there were issues with C++ codes if our main was always in Fortran.

As for parsing command line arguments in Fortran, in standard Fortran2003+ you'd do something like this:

program main

integer :: i, argc, ierr
integer, parameter :: arg_len = 256
character(len=arg_len), dimension(:), allocatable :: raw_arguments

argc = command_argument_count()
allocate(raw_arguments(argc))
do i = 1, argc
    call get_command_argument(i, raw_arguments(i), arg_len, ierr)
end do

call charm_init(argc, raw_arguments)

end program main

Pre-Fortran2003, all compilers (at least the ones we care about: gfortran, ifort, xlf, cray f77, pgf77) implemented the same GNU extension APIs for parsing the command line, iargc() and getarg(), which function similarly to the Fortran2003 standard ones.

[evan-charmworks]

Original date: 2018-05-17 22:08:28

---

I cobbled this together:

args.f90:

program main

use iso_c_binding, only: C_CHAR, C_PTR, C_LOC
implicit none
integer :: i, argc, ierr
integer, parameter :: arg_len = 256
character(kind=c_char, len=arg_len), dimension(:), allocatable, target :: raw_arguments
TYPE(C_PTR), DIMENSION(:), allocatable :: argv

write (*, '(a)') "-- Fortran --"

argc = command_argument_count()+1
write (*, '(a, I8)') "argc = ", argc

allocate(raw_arguments(argc))
allocate(argv(argc))
do i = 1, argc
    call get_command_argument(i-1, raw_arguments(i))
end do

write (*, '(a)') "argv:"
do i = 1, argc
    write (*, '(a)') raw_arguments(i)
end do

do i = 1, argc
    raw_arguments(i) = TRIM(ADJUSTL(raw_arguments(i)))//char(0)
end do

do i = 1, argc
    argv(i) = C_LOC(raw_arguments(i))
end do

call charm_init_wrapper(argc, argv)

end program main

charm_init.cpp:

#define TESTGLOBALS 0

#if TESTGLOBALS
struct MyStruct
{
  int x;
  MyStruct() { x = 1; }
  MyStruct(int y) { x = y; }
};

static MyStruct MyGlobal1;
static MyStruct MyGlobal2{2};
MyStruct MyGlobal3{3};
#endif

#include <iostream>

static void charm_init(int argc, char **argv)
{
  std::cout << "-- C++ --" << std::endl;

#if TESTGLOBALS
  // test whether C++ constructors of globals executed correctly
  static MyStruct MyGlobal4{4};
  std::cout << "MyGlobal1 = " << MyGlobal1.x << std::endl;
  std::cout << "MyGlobal2 = " << MyGlobal2.x << std::endl;
  std::cout << "MyGlobal3 = " << MyGlobal3.x << std::endl;
  std::cout << "MyGlobal4 = " << MyGlobal4.x << std::endl;
  std::cout << std::endl;
#endif

  std::cout << "argc = " << argc << std::endl;
  std::cout << "argv:" << std::endl;

  for (int i = 0; i < argc; ++i)
    std::cout << argv[i] << std::endl;
}

extern "C" void charm_init_wrapper_(int *argc, char **argv)
{
  charm_init(*argc, argv);
}

terminal:

$ gfortran-8 -c args.f90 -o args.o -g3
$ g++-8 -c charm_init.cpp -o charm_init.o -g3
$ g++-8 charm_init.o args.o -o argstest -lgfortran
$ ./argstest 1 2 3
-- Fortran --
argc =        4
argv:
./argstest
1
2
3
-- C++ --
argc = 4
argv:
./argstest
1
2
3

With TESTGLOBALS defined to 1 I also confirmed that C++ constructors of globals are executed correctly.

[evan-charmworks]

Original date: 2018-05-18 21:52:27

---

https://charm.cs.illinois.edu/gerrit/4203 https://github.com/UIUC-PPL/charm/commit/9aa1e4c26980b3a6cd29d8a73bda681d19f11b70