search

ctuning / reproduce-milepost-project

Collective Knowledge workflow for the MILEPOST GCC (machine learning based compiler). See how it is used in the collaborative project with the Raspberry Pi foundation to support collaborative research for multi-objective autotuning and machine learning techniques, and prototype reproducible papers with portable workflows:
http://cKnowledge.org/rpi-crowd-tuning
GNU General Public License v2.0
47 stars 6 forks source link

Failure to extract features for MPI programs #7

Closed twang15 closed 6 years ago

twang15 commented 6 years ago

Feature extraction for MPI programs in c language fails with the following message:

mpicc -o amg_linklist.o -c -I.. -v -O3 --ct-extract-features -frounding-math -fsignaling-nans -DTIMER_USE_MPI -DHYPRE_USING_OPENMP -DHYPRE_LONG_LONG -DHYPRE_TIMING amg_linklist.c mpicc for MVAPICH2 version 2.2 Using built-in specs. Target: x86_64-unknown-linux-gnu Configured with: ../milepost-gcc-4.4.4/configure --prefix=/home/twang15/CK-TOOLS/gcc-milepost-4.4.4-linux-64 --enable-languages=c,c++,fortran --disable-multilib --with-gmp=/home/twang15/CK-TOOLS/gmp-5.0.5-linux-64 --with-mpfr=/home/twang15/CK-TOOLS/mpfr-3.1.0-linux-64 --with-mpc= Thread model: posix gcc version 4.4.4 (GCC) COLLECT_GCC_OPTIONS='-o' 'amg_linklist.o' '-c' '-I..' '-v' '-O3' '-fct-extract-features' '-frounding-math' '-fsignaling-nans' '-DTIMER_USE_MPI' '-DHYPRE_USING_OPENMP' '-DHYPRE_LONG_LONG' '-DHYPRE_TIMING' '-I/opt/ohpc/pub/mpi/mvapich2-gnu/2.2/include' '-mtune=generic' /home/twang15/CK-TOOLS/gcc-milepost-4.4.4-linux-64/libexec/gcc/x86_64-unknown-linux-gnu/4.4.4/cc1 -quiet -v -I.. -I/opt/ohpc/pub/mpi/mvapich2-gnu/2.2/include -DTIMER_USE_MPI -DHYPRE_USING_OPENMP -DHYPRE_LONG_LONG -DHYPRE_TIMING amg_linklist.c -quiet -dumpbase amg_linklist.c -mtune=generic -auxbase-strip amg_linklist.o -O3 -version -fct-extract-features -frounding-math -fsignaling-nans -o /tmp/ccRaiVDZ.s cc1: error: unrecognized command line option "-fct-extract-features"

twang15 commented 6 years ago

It seems to me that ctuning-cc/c++ is a wrapper around milepost gcc/g++. If this is true, then the root cause for this error is that mpicc should invoke ctuning-cc first, not directly invoking cc1.

The invocation relation should look like this: mpicc -> ctuning-cc ->milepost gcc ->cc1

But the current invocation is like: mpicc-> milepost gcc ->cc1.

But, milepost gcc does not understand feature extraction option "--ct-extract-features" while ctuning-cc does.

twang15 commented 6 years ago

The challenges is that mpicc does not know the existence of ctuning-cc and thus has no way to invoke it.

To address this challenge, we can rename ctuning-cc as gcc, so that mpicc will invoke it. However, the fact that ctuning-cc invokes milepost gcc with name as gcc prevents us from doing this. The solution seems to be:

Solution 1:

  1. rename ctuning-cc as gcc, so that mpicc will invoke it.
  2. rename milepost gcc's binary as milepost-gcc.
  3. modify ctuning-cc's source code to invoke milepost-gcc.

Solution 2:

  1. Extract mpicc's flags, which includes include/header paths (in the form as -Ixx), link paths (in the form as -Lxx for dynamic library, -lxx for static library)
  2. Use ctuning-cc to replace mpicc as the compilation driver.
twang15 commented 6 years ago

@gfursin

twang15 commented 6 years ago

The solution turns out to be very simple.

  1. copy system mpicc to my local directory $HOME/benchmark. (because I want to modify it, but I do not have root access)
  2. Modify local copy mpicc, replace CC=gcc to CC=$(CK_CC)
  3. export PATH=$HOME/benchmark:$PATH.
  4. Everything as before #4 #5 #6
gfursin commented 6 years ago

You are right, @twang15 - ctuning-cc is a wrapper around MILEPOST GCC, so your solution is correct! Thanks a lot for figuring it out and reporting. I also added this info to MILEPOST readme!!!