search

team-pltb / pltb

https://mhoff.net/blog/2016/06/17/pltb/
GNU General Public License v2.0
1 stars 1 forks source link

Statically linked binaries #4

Open mhoff opened 9 years ago

mhoff commented 9 years ago

We should provide statically linked binaries for users who just want to use our application as convenient as possible. @stefanorf, @briehm I think somebody already looked into this issue. If yes, could you please provide your findings. If not, please also reply to this issue so that we can plan & assign this issue appropriately.

briehm commented 9 years ago

We first need to statically compile the MPI library, i.e. 

./configure --without-memory-manager --without-libnuma --enable-static --disable-shared

http://www.open-mpi.de/faq/?category=mpi-apps#static-mpi-apps

After installing MPI and PLL, I tried to build a static version of our code via:

mpicc <Obj_files.o> -static -Wl,--start-group -lpll-sse3 -lmpi -Wl,--end-group -lm -lrt

Then I got tons of errors of the following kind:

<some_file.c:(.text+0x42)> undefined reference to `some_variable / function'
<some_lib.a:(.text+0x42)> In function `some_function': undefined reference to `some_variable / function'

This kind of problem often occurs, if the libraries are in the wrong order, but with the linking option --start-group/end-group this should not be the case. http://stackoverflow.com/questions/45135/why-does-the-order-in-which-libraries-are-linked-sometimes-cause-errors-in-gcc

mhoff commented 9 years ago

First of all, isn't it enough to statically link PLL and dynamically link MPI? Is this even possible? @stamatak what is your opinion regarding this issue?

https://www.open-mpi.org/faq/?category=mpi-apps#static-mpi-apps

Fully static linking is not for the weak, and it is not recommended. But it is possible, with some caveats.

Nevertheless I will now also try to statically link both libraries (as this is the most convenient way of providing our application). But I doubt, that this can be very efficient in relation to a dynamically linked MPI implementation in the OS.

mhoff commented 9 years ago

Hybrid linking (static PLL, shared MPI) should be possible using: LFLAGS=-Wl,-Bstatic -lpll-sse3 -Wl,-Bdynamic $(shell mpicc --showme:link) -lm -lrt The resulting binary has a size of 2.0M and the 'normal' one of 40K. Also ldd shows no sign of PLL:

$ ldd pltb.out
linux-vdso.so.1 =>  (0x00007ffe94c23000)
libmpi.so.1 => /usr/lib/libmpi.so.1 (0x00007efec6272000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efec5f6c000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007efec5d64000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efec5b46000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efec5782000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007efec557f000)
libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5 (0x00007efec533f000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007efec5135000)
/lib64/ld-linux-x86-64.so.2 (0x00007efec65f3000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007efec4f2a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efec4d26000)

@briehm Can you verify this on your Darwin-based system?

I am just now evaluating how to integrate this configuration with mpicc in a correct manner. Calling mpicc like before does not sound right as we are then supplying two linking configurations to gcc. In fact it seems to work that way, but I don't have a good feeling about this.

On my system mpicc wraps the following call: gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc which is nearly equal to mpicc --showme:command` `mpicc --showme:compile` `mpicc --showme:link producing gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc Note the redundant -pthread. Hence, replacing the call to mpicc with the above command, while substituting the "link" part with our custom configuration produces what we want with an extra -pthread. Opinions on that?

mhoff commented 9 years ago

I pushed the mixed version (ff74183).

stamatak commented 9 years ago

thanks, in any case I guess a statically linked sequential and PThreads based versionshould be sufficient for most users, I didn't think about MPI issues

alexis

On 14.06.2015 20:14, Michael Hoff wrote:

I pushed the mixed version (ff74183 https://github.com/team-pltb/pltb/commit/ff741831b3c1b09c619a536dba8cd861ada1c392).

— Reply to this email directly or view it on GitHub https://github.com/team-pltb/pltb/issues/4#issuecomment-111860057.

Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University of Arizona at Tucson

www.exelixis-lab.org

briehm commented 9 years ago

One needs to alter the linker flags for OS X, because it does not use the GNU linker and thus does not have the Bstatic and Bdynamic options: LFLAGS=-L/usr/local/lib/static -lpll-sse3 -Wl,-dynamic $(shell mpicc --showme:link) -lm Prior one needs to move the static libraries into a dedicated folder, such that the linker does not use the dynamic libs instead (it looks for dynamic libs first, then static libs afterwards..) Interestingly, the resulting binary is 706K in size.

mhoff commented 9 years ago

Did you verify the correctness of your process with ldd? Is there no -Wl,-static variant for OS X? Does the folder name imply the kind of linking? This solution seems weird to me.

briehm commented 9 years ago

Yes, I checked this with the program otool (similar tool to ldd). 

/opt/local/lib/mpich-mp/libmpi.12.dylib (compatibility version 13.0.0, current version 13.5.0)
/opt/local/lib/mpich-mp/libpmpi.12.dylib (compatibility version 13.0.0, current version 13.5.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/opt/local/lib/libgcc/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

If someone wants to compile our program, he would need MPI and GCC installed. The PLL is linked statically (otherwise it would be printed out in the list above).

No, apparently there doesn't exist a similar option. But as I just figured out, you can do something else instead:

MPICH_CC=gcc OMPI_CC=gcc mpicc src/frontend.o src/ic.o src/masterworker.o src/models.o \
src/mpi_backend.o src/pltb.o src/pltb_frontend.o src/sequential.o \
/PATH/TO/libpll-sse3.a -lm -o pltb.out

So you just need to give the static library file path as a parameter to the compiler, just like some object file. Then it should link to that static library instead of the dynamic one.

mhoff commented 9 years ago

If someone wants to compile our program, he would need MPI and GCC installed. The PLL is linked statically (otherwise it would be printed out in the list above).

Do you mean "execute"? Therefore we can provide partially linked binaries for MAC and Linux based systems?

thanks, in any case I guess a statically linked sequential and PThreads based versionshould be sufficient for most users, I didn't think about MPI issues

This way completely statically linked binaries could be produced. @briehm do you want to test this? In frontend.c there is a switch #define MPI_MASTER_WORKER 1 which should - hopefully - completely disable MPI includes, if turned off.

briehm commented 9 years ago

@mhoff: Yes, if we provide the binaries, then they would need MPI and GCC in order to execute our program. Sorry for the misunderstanding!

In frontend.c there is a switch #define MPI_MASTER_WORKER 1 which should - hopefully - completely disable MPI includes, if turned off.

I've tested it, but it did not work. The compiled binary still had the mpi dynamic libs as dependencies.

mhoff commented 9 years ago

I've tested it, but it did not work. The compiled binary still had the mpi dynamic libs as dependencies.

Did you also compile without mpicc?

stamatak commented 9 years ago

I think we should not worry too much about pre-compiled MPI version binaries, whoever can use an MPI code is probably also able to compile it,

alexis

On 30.06.2015 07:48, Michael Hoff wrote:

I've tested it, but it did not work. The compiled binary still had
the mpi dynamic libs as dependencies.

Did you also compile /without/ |mpicc|?

— Reply to this email directly or view it on GitHub https://github.com/team-pltb/pltb/issues/4#issuecomment-116981054.

Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University of Arizona at Tucson

www.exelixis-lab.org

mhoff commented 9 years ago

Thats probably true. However I just applied some minor changes which enables us to build completely mpi-independent binaries. With Makefile.nompi and #define MPI_MASTER_WORKER 0 everything should work out of the box. @briehm can you verify this? The only thing left to do is to combine Makefile.nompi and Makefile.static_call to produce 100% static (sequential) binaries.

stamatak commented 9 years ago

thanks :-)

alexis

On 30.06.2015 22:00, Michael Hoff wrote:

Thats probably true. However I just applied some minor changes which enables us to build completely mpi-independent binaries. With |Makefile.nompi| and |#define MPI_MASTER_WORKER 0| everything should work out of the box. @briehm https://github.com/briehm can you verify this? The only thing left to do is to combine |Makefile.nompi| and |Makefile.static_call| to produce 100% static (sequential) binaries.

— Reply to this email directly or view it on GitHub https://github.com/team-pltb/pltb/issues/4#issuecomment-117322424.

Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University of Arizona at Tucson

www.exelixis-lab.org

mhoff commented 9 years ago

I just merged the Makefiles and published a release (b6090846e146). Note that building against pthreads versions is not supported (yet). The release contains two binaries for linux based systems. @stefanorf could you please test those binaries? @briehm can you provide binaries for OS X?

stamatak commented 9 years ago

thanks :-)

alexis

On 03.07.2015 20:40, Michael Hoff wrote:

I just merged the Makefiles and published a release (b609084 https://github.com/team-pltb/pltb/commit/b6090846e1467749ae5fcdd3abadbb2ef29b4c37). Note that building against pthreads versions is not supported (yet). The release contains two binaries for linux based systems.

— Reply to this email directly or view it on GitHub https://github.com/team-pltb/pltb/issues/4#issuecomment-118407617.

Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University of Arizona at Tucson

www.exelixis-lab.org