Speed up cross-compilation

[Diaoul]

We'd save a lot of time cross compiling by using ccache. I'll create a branch with an implementation to try it out.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[cytec]

sounds pretty good

[cytec]

@Diaoul i've just came across this again and thought id test it. changed: https://github.com/SynoCommunity/spksrc/blob/master/mk/spksrc.tc.mk#L65 to:

echo TC_ENV += `echo $${target} | tr [:lower:] [:upper:] `=\"ccache $(WORK_DIR)/$(TC_BASE_DIR)/bin/$(TC_PREFIX)-$${source}\" ; \

and got huge performance increases! is this still valid?

[cytec]

UPDATE: just tried to compile libtheora which doesn't work with the approach from above as it tries to run:

    /bin/bash ../libtool --mode=compile ccache /home/cytec/syno/toolchains/syno-armada375-5.1/work/armv7-marvell-linux-gnueabi-hard/bin/arm-marvell-linux-gnueabi-gcc -DHAVE_CONFIG_H -I. -I. -I.. -I../include  -I/home/cytec/syno/spk/ffmpeg/work-armada375-5.1/install//usr/local/ffmpeg/include  -I/home/cytec/syno/spk/ffmpeg/work-armada375-5.1/install/usr/local/ffmpeg/include    -Wall -Wno-parentheses -O3 -fforce-addr -fomit-frame-pointer -finline-functions -funroll-loops  -I/home/cytec/syno/spk/ffmpeg/work-armada375-5.1/install//usr/local/ffmpeg/include  -c -o apiwrapper.lo `test -f 'apiwrapper.c' || echo './'`apiwrapper.c

[cytec]

@Diaoul so i've played a little bit with this, and prefixing the compiler command isn't really a way here as there are some tools which call the compiler seperate (like libtheora for example) and those are failing/stopping to run if i'll prefix the command with ccache...

so the way to go might be creating symbolic links: https://ccache.samba.org/manual.html#_run_modes i've played with them a little but didn't really get them to work. Seems like the compiler bust be in PATH and also i was unable to provide a special binary like this /home/cytec/syno/toolchains/syno-armada375-5.1/work/armv7-marvell-linux-gnueabi-hard/bin/arm-marvell-linux-gnueabi-ccache-gcc which i would prefere, so one can disable ccache for some packages easy if it would be needed.

maybe you or @Dr-Bean can have a look? as this is a massive improve when compiling stuff more than once (mostly when testing and for really common depends)

[cytec]

Seems like my previous post won't work :( but an alternative may be when setting up the toolchain we can link the compiler to ccache and add the TC_PATH to the users PATH...

[cytec]

@Diaoul @Dr-Bean ok... i've got a new Idea:

1. when setting up the toolchains (example for x64) create 2 symlinks:

   x86_64-pc-linux-gnu-ccache-g++ -> x86_64-pc-linux-gnu-g++ x86_64-pc-linux-gnu-ccache-gcc -> x86_64-pc-linux-gnu-gcc

2. then add a symlink for ccache in /usr/lib/ccache:

   x86_64-pc-linux-gnu-ccache-g++ -> ../../bin/ccache x86_64-pc-linux-gnu-ccache-gcc -> ../../bin/ccache

3. change tc.mk to not use path for those binaries and change PATH like this (sorry don't have much experience in mk files):

[...]
TOOLS = ld cpp nm as ranlib ar strip objdump readelf
CCACHE_TOOLS = ldshared:"gcc -shared" cc:gcc cxx:g++
[...]
@for tool in $(TOOLS) ; \
    do \
      target=`echo $${tool} | sed 's/\(.*\):\(.*\)/\1/'` ; \
      source=`echo $${tool} | sed 's/\(.*\):\(.*\)/\2/'` ; \
      echo TC_ENV += `echo $${target} | tr [:lower:] [:upper:] `=\"$(WORK_DIR)/$(TC_BASE_DIR)/bin/$(TC_PREFIX)-$${source}\" ; \
    done
    @for tool in $(CCACHE_TOOLS) ; \
    do \
      target=`echo $${tool} | sed 's/\(.*\):\(.*\)/\1/'` ; \
      source=`echo $${tool} | sed 's/\(.*\):\(.*\)/\2/'` ; \
      echo TC_ENV += `echo $${target} | tr [:lower:] [:upper:] `=\"$(TC_PREFIX)-ccache-$${source}\" ; \
    done
@echo TC_ENV += PATH=/usr/lib/ccache/:$$PATH:$(WORK_DIR)/$(TC_BASE_DIR)/bin/
[...]   

when tc-env is created use those binaries and also change PATH to /usr/lib/ccache/:$$PATH:/$(WORK_DIR)/$(TC_BASE_DIR)/bin/ which produces something like this:

env PKG_CONFIG_LIBDIR=/home/cytec/syno/cross/nzbget/work-x64-5.1/install//usr/local/lib/pkgconfig WORK_DIR=/home/cytec/syno/cross/nzbget/work-x64-5.1 INSTALL_PREFIX=/usr/local TC=syno-x64-5.1  LD="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ld" CPP="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-cpp" NM="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-nm" AS="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-as" RANLIB="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ranlib" AR="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ar" STRIP="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-strip" OBJDUMP="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-objdump" READELF="/home/cytec/syno/toolchains/syno-x64-5.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-readelf" LDSHARED="x86_64-pc-linux-gnu-ccache-gcc -shared" CC="x86_64-pc-linux-gnu-ccache-gcc" CXX="x86_64-pc-linux-gnu-ccache-g++" CFLAGS=" -I/home/cytec/syno/cross/nzbget/work-x64-5.1/install//usr/local/include " CPPFLAGS=" -I/home/cytec/syno/cross/nzbget/work-x64-5.1/install//usr/local/include " CXXFLAGS=" -I/home/cytec/syno/cross/nzbget/work-x64-5.1/install//usr/local/include " LDFLAGS=" -L/home/cytec/syno/cross/nzbget/work-x64-5.1/install//usr/local/lib -Wl,--rpath-link,/home/cytec/syno/cross/nzbget/work-x64-5.1/install//usr/local/lib -Wl,--rpath,/usr/local/lib " ./configure  --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/usr/local --with-shared --enable-rpath --without-manpages --without-tests --enable-overwrite --disable-home-terminfo --with-default-terminfo-dir=/home/cytec/syno/cross/nzbget/work-x64-5.1/install//usr/local/share/terminfo --with-terminfo-dirs=/usr/local/share/terminfo

that seems to work for me... maybe there should be an option to enable/disable ccache by skipping the 3rd step...

as is already said: i don't have much experience with this stuff and most of it was hacked in there somehow (created the symlinks manually etc) but maybe @Dr-Bean can help :)

[Dr-Bean]

Don't know about that last sentence there, I'm not familiar with ccache at all. I'll have to see if I can do something about that, but at this point, I probably have more questions than you :P

[cytec]

im also not really familiar with ccache... there was a lot of try and error involved... but i've came up with something like this (which seems to work): https://github.com/cytec/spksrc/commit/b335595735318b3d610cd6a09775f4ed42b68677

which seems to work and also if DISABLE_CCACHE is set wont use the linked binaries... still there is the PATH being set... idk how to fix that and maybe the whole thing i wrote is not really "clean" and probably can be improved... but you should get the basic functionality ;)

[cytec]

@Dr-Bean @Diaoul any input on this?

[Dr-Bean]

Haven't really checked yet :/ Based on your previous comments, this works? I'm probably missing something wrt the PATH changes needed, but I can't figure it out by just looking at it, I'll need to test it.

One comment to consider is to set it up like this, makes for less repeating:

TOOLS = ld cpp nm as ranlib ar strip objdump readelf ldshared:"gcc -shared" 
ifeq ($(strip $(DISABLE_CCACHE)),)
CCACHE_TOOLS = cc:gcc cxx:g++
else
TOOLS += cc:gcc cxx:g++
endif