Using Llamacc/Llamac++ with -gsplit-dwarf flag

[esneyder2]

I've been trying to build our embedded application using llamac++. My compilations work fine and I'm able to get all the corresponding object (.o) files. However, our build system uses the "-gsplit-dwarf" flag to split the debug symbols into a separate .dwo file. (More info on the split-dwarf flag: https://www.productive-cpp.com/improving-cpp-builds-with-split-dwarf/.). This means that every compilation step is supposed to produce a corresponding .o file and a .dwo file. After the build system links the object files to create the application binary, it runs the dwp tool (from binutils) to wrap up all the .dwo files into a single .dwp file that gdb can use to load debug info. Llamac++ however does not return any of the .dwo files from the cloud compilation and hence the final step of our build system fails as the dwp tool is unable to find any .dwo files.

I tried to modify the llamacc source code to return .dwo files as well, but I haven't had any success with it. These are the steps I've gone through:

1. Added code line to “args.go” (specify which line number in args.go) to generate .dwo files as an output

out.Output = out.Output + " " + replaceExt(out.Input, ".dwo")

2. Rebuilt llama binaries using go install ./... at the root of the repo
3. Ran my compilation using llamac++

The error message that I’m getting is:

FAILED: libBOS_ppc.a.p/BOS_source_Test_PthreadTest.cpp.o 
/usr/bin/env LLAMACC_LOCAL_CXX=/opt/poky/2.5.3/ppc/sysroots/x86_64-pokysdk-linux/usr/bin/powerpc-poky-linux/powerpc-poky-linux-g++ LLAMACC_FUNCTION=gcc_test3 llamac++ -mtune=e300c3 --sysroot=/opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux -IlibBOS_ppc.a.p -I. -I.. -I../BOS/include -I../BOS/include/BOS/Test -I../BOS/platform/linux/include -I../Util/include -I../UnitTest/include -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -fno-exceptions -ffunction-sections -fdata-sections -fno-working-directory -Werror=poison-system-directories -Wall -std=c++17 -Wnon-virtual-dtor -fdiagnostics-color=always -g -gsplit-dwarf -DLINUX -DLABRINTH -DNDEBUG -Wno-unused -Wno-uninitialized -O2 -MD -MQ libBOS_ppc.a.p/BOS_source_Test_PthreadTest.cpp.o -MF libBOS_ppc.a.p/BOS_source_Test_PthreadTest.cpp.o.d -o libBOS_ppc.a.p/BOS_source_Test_PthreadTest.cpp.o -c ../BOS/source/Test/PthreadTest.cpp
Usage: /opt/poky/2.5.3/ppc/sysroots/x86_64-pokysdk-linux/usr/libexec/powerpc-poky-linux/gcc/powerpc-poky-linux/7.3.0/objcopy [option(s)] in-file [out-file]
Copies a binary file, possibly transforming it in the process

The error message is confusing because we don't use the objcopy tool in our build system, even though it is present in our SDK. Would appreciate your help in trying to get this to work as we use the split-dwarf flag for all our application builds across multiple processor architectures.

[nelhage]

https://github.com/nelhage/llama/pull/57 fixes in my local testing. Please let me know if it doesn't work for you.

[esneyder2]

Thank Nelhage for helping us with this. I've pulled down your latest changes and I’m now able to get all the object (.o) files and (.dwo)files for my compilation. However there still remains a complication; when running the dwp tool to collect all the dwo files, I'm getting this error:

[6513/6513] Generating LabrinthDevice_ppc with a custom command
FAILED: LabrinthDevice_ppc LabrinthDevice_exe.dwp LabrinthDevice_ppc.map
/opt/poky/2.5.3/ppc/sysroots/x86_64-pokysdk-linux/usr/bin/powerpc-poky-linux/powerpc-poky-linux-dwp -e /home/esneyder/src/sira-firmware/.ppcbuild5/LabrinthDevice_exe && /opt/poky/2.5.3/ppc/sysroots/x86_64-pokysdk-linux/usr/bin/powerpc-poky-linux/powerpc-poky-linux-strip --strip-unneeded /home/esneyder/src/tesira-firmware/.ppcbuild5/LabrinthDevice_exe -o LabrinthDevice_ppc
/opt/poky/2.5.3/ppc/sysroots/x86_64-pokysdk-linux/usr/bin/powerpc-poky-linux/powerpc-poky-linux-dwp: error: cannot open _root/home/esneyder/src/tesira-firmware/.ppcbuild5/LabrinthDevice_exe.p/meson-generated_.._LabrinthDevice_meson_gen_IEEE8021X_data.cpp.dwo: No such file or directory
/opt/poky/2.5.3/ppc/sysroots/x86_64-pokysdk-linux/usr/bin/powerpc-poky-linux/powerpc-poky-linux-dwp: fatal error: _root/home/esneyder/src/tesira-firmware/.ppcbuild5/LabrinthDevice_exe.p/meson-generated_.._LabrinthDevice_meson_gen_IEEE8021X_data.cpp.dwo: can't open

ninja: build stopped: subcommand failed.

It likes the object files that holds the info for the location of the .dwo file is pointing to the location of the .dwo in the lambda container that runs the compilation (hence the '_root' prefix to the .dwo file that cant be found). Do you have any ideas on how we could tackle this issue?

[nelhage]

Ugh, that's annoying. #55 attempts to fix this with -fdebug-prefix-map, which is intended to tell the compiler to rewrite paths to point to the right ones locally, but clearly I missed a path or otherwise didn't get it all the way there. I'll try to take a look further when I have a chance…

[njpau]

Hi @nelhage , should we create a separate issue to address the above issue or do you just want to re-open this task?

[nelhage]

(Looking into this now after getting back from vacation)

Ugh, this looks to be an upstream GCC bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91888

It seems there's no way to get -gsplit-dwarf to change the path it writes to the DWO. I'm not sure there's an easy workaround on our end. It seems ccache has a similar bug: https://github.com/ccache/ccache/discussions/591

From that discussion, one possible workaround is tweaking your build system to use dwp to split the debug symbols post-facto; I don't know if that's a possibility in your environment.