search

AOSC-Archive / autobuild3

AOSC OS package maintenance toolkit (version 3)
https://aosc.io
GNU General Public License v2.0
24 stars 17 forks source link

Debug archives #29

Open Artoria2e5 opened 9 years ago

Artoria2e5 commented 9 years ago

We need to specify a way to make debug archives. Currently we are quite sure that in some earlier process, like in stripping, we will dump the debug stuffs to /usr/lib/debug/path/to/orig/elf.dbg, but we are not yet sure how we should dump the symbols.

There are currently two ways of making debugs. We can actually make them in parallel.

if [ "$CROSS" ]; then
  abtryexe eu-make-debug-archive &&
  # Haven't figured out how eu-make-debug-archive works yet.
else
  # debhelper: dh_debug (perl): sub make_debug
  abtryexe objcopy &&
  dump_debug(){ objcopy --only-keep-debug --compress-debug-sections "$SRCDIR"{,/usr/lib/debug}/"$1"; }
fi || abicu "No candidate for dump_debug." || dump_debug(){ true; }

Issue Metadata

Depends: #7

Artoria2e5 commented 9 years ago

Well we really need to figure out how the eu- magic works and if objcopy works with static archives.

Artoria2e5 commented 9 years ago

We need to figure out which debug level we are going to use.

For everyday use I may say -g1, but since we are splitting the package, I would say the debug size doesn't matter that much, so maybe -g3?

We can use a #7-like strategy to produce a package called PKGNAME-fulldbg with all -g3 debugs not stripped, lots of Debugging Options like -fsanitize enabled (unluckily there's no -fsanitize=all) and uses the debug CFLAGS. It should be Conflict-Replace. It's for really crazy debugging lovers and we are not going to provide this in our binary repo. Just a shorthand for manual building.

It's also said that gold #30 doesn't support dedups in debugging strings and we need to verify it.

Oh, the stragegy spelling is another unrool-loops for me.

Artoria2e5 commented 9 years ago

A few more notes on gold and symbols:

MingcongBai commented 9 years ago

@Arthur2e5

First of all, on the original issue message. Elfutils, I suppose, once being used, it should be used everywhere, all platforms.

On CFLAGS, yes, LTO should be cleaned away too, onto the debug distribution.

On debug level, IMHO, if we are meant to build a debug package, why don't we just maximize its capacity to be debugged?

MingcongBai commented 9 years ago

@Arthur2e5

Also, on GDB part, GDB is not even close to be the exclusive debugger choice for AOSC OS. So no, please do not strip away stuff unneeded by GDB.

Again, if it is meant for debugging, give it the full right and capacity.

Artoria2e5 commented 9 years ago

@MingcongBai The problem is that an optimized package (unless with -Og) may not be that suitable for deep debugging, so I'm questioning on this.

I'm suggesting to keep the LTO stuffs in static libraries where they may be used for speeding up the stuffs. Honestly I don't know exactly how we can extract them using objcopy (maybe by extracting all segs called LTO?), and neither do I know what will the traditional ld do with those LTO segs in static libs (ld doesn't have a flag to throw away LTO segs.)

The problem that the CCs and LDs don't know how to get the symbols back from somewhere else can be solved by using some eu-unstrip magic in PKGNAME-lto, so I suppose it won't be a problem here.

By the way, clang also supports gold plugins, but I don't think LLVM IRs and GIMPLE are stored compatibly to each other in LTOed archives. Ahhhhhhhhhhhhh! Well, I think I should strip them out where possible.

Wait, were we talking about LTO and debugging? I forgot to mention that whole-program optimizations like LTO can make detailed debugging info added in -g3 almost useless.

MingcongBai commented 9 years ago

@Arthur2e5

When you link LTO objects, gold should be used anyways, the change is acceptable in most cases. Some edge cases like chromium fails to build with Gold as it simply overfeeds the linker.

And by your case, many sub-cases will be needed for your implementation, and possibly a dual build scenario too. Although, regarding dual or even more passes in build may become unbearably time consuming, for instance, LibreOffice (which can take hours simply for a build with parallelism enabled on our public bot).

Artoria2e5 commented 9 years ago

@MingcongBai Suggesting this:

CFLAGS_COMMON_NOLTO+='-g3 '
CFLAGS_COMMON_LTO+='-g1 '

For serious debugging we are going to use NOLTO=1 autobuild.

MingcongBai commented 9 years ago

@Arthur2e5

However, should we ship debug packages or simply leave them for users/devs to self-service?

Artoria2e5 commented 9 years ago

@MingcongBai serious debugging should be self-serviced. Still considering -g1 as the best option.

BTW I don't think I will do anything like a dual build. Since most of our packages are built with CLANG, only packing LTO hints once should be enough — it would be quite rare (in a sence of frequency) to meet some mixups. And how do you guarantee that mixed LTO hints will work? They point to different object files!

An extra problem with clang ltos. http://llvm.org/docs/GoldPlugin.html tells me that I should mask ranlib since it doesn't support LLVM bitcodes, and it's quite likely the case I have been getting on OS X wherer I fltoed bash with static readline.

MingcongBai commented 9 years ago

@Arthur2e5

Unfortunately, no, Clang is not the default and will not be until we can meet in July and we will decide on that.

And again, no. LTO is not always available as many, and I mean many packages fails to build, or have regressed performance or stability in its binary form. Further experiment will be conducted soon.

Artoria2e5 commented 9 years ago

@MingcongBai It seems that we should do ! bool NOSTATIC && ! bool NOLTO || NOCLANG=1 since Clang loves replacing regular objects with LLVM BCs into static libraries which breaks the builds. GIMPLE segments seems to be simply additional segs to normal objects so I currently consider it safe.

Artoria2e5 commented 9 years ago

Well, first of all we should add the following segments to LLVM and binutils' defines:

# mysterious var ARCH_EXE_FORMAT: ELF/PE/COFF/a.out/etc.. (note that PE is an
# extension of COFF and in COFF test funcs we should consider blah blah blah)
# and a never-implemented support for things like autobuild_ELF -- Toooooo silly.

# This is for binutils
# expected already --enable-plugin --enable-lto
if [ "$ARCH_EXE_FORMAT" == ELF ]; then # MAGIC!
  AUTOTOOLS_AFTER+=' --enable-ld=default --enable-gold'
else
  rm -rf $SRCDIR/autobuild/subpkgs/-dev # Includes for later use with llvm
fi

# And LLVM
if [ "$ARCH_EXE_FORMAT" == ELF ]; then # MAGIC AGAIN
  BUILDDEP+=' binutils-dev'
  AUTOTOOLS_AFTER+=' --with-binutils-include=/usr/lib/include'
fi

And still another question.. It seems that clang LTOs even without gold, and I don't know if clang can accept an option so it doesn't put plain LLVM BCs into statics even if gold, nm and ar supports that.

Artoria2e5 commented 9 years ago

My English reading ability seems to be dropping a lot. Read http://llvm.org/docs/GoldPlugin.html and http://llvm.org/docs/LinkTimeOptimization.html lots of times but still unable to figure out what clang would do to statics with gold-based LTO.

GCC is still safe, as shown in a TODO in https://gcc.gnu.org/wiki/LinkTimeOptimization .

Currently, GIMPLE is generated internally by the compiler and emitted inside special sections as binary content. This permits the LTO front end (lto1) read and reconstitute the internal representation that was generated from the original source code.

Btw for huge stuffs like chrome, the GCC doc tells us to use the scalable -fwhopr instead: https://gcc.gnu.org/projects/lto/whopr.pdf .

Needs more sleep.

Artoria2e5 commented 9 years ago

https://gcc.gnu.org/wiki/LTO_Debug is telling me that GCC, like clang in my imagination, stores an intermediate representation (IR) of the code in the object file rather than compiled object code (the LGEN phase). Investigating if autotools would go back to LGEN those IRs correctly. This is also important for clang.

Running fsck in-brain and found lots of orphaned inodes. Fsck.

Artoria2e5 commented 9 years ago

http://stackoverflow.com/questions/25878407/how-can-i-use-lto-with-static-libraries show that LTO-static is really some crazy combination, unless supplied with the correct plugins. From this I guess there is no backwards LGEN at all. (Therefore discarding LTOs in statics almost means discarding everything?)

Still don't know if the plugins can help me link GCC and CLANG LTO statics together.

Artoria2e5 commented 9 years ago

Maybe ln-ing both LLVMGold.so and liblto_plugin.so into /usr/lib/bfd-plugins can help us. This is already mentioned in http://llvm.org/docs/GoldPlugin.html but I missed it. ArchLinux FS28479 says the same.

Unluckily I can't test this before I expand an OS3 tarball into someone's Ubuntu 14.04 Minecraft Server.

Btw I saw dragonegg-4.8 in Ubuntu. @Icenowy it seems that people can still use a ***\ even if ze has a boyfriend.

MingcongBai commented 9 years ago

It is there already. 2015年6月25日 下午11:33于 "Mingye Wang" notifications@github.com写道：

Maybe ln-ing both LLVMGold.so and liblto_plugin.so to /usr/lib/bfd-plugins can help us.

— Reply to this email directly or view it on GitHub https://github.com/AOSC-Dev/autobuild3/issues/29#issuecomment-115294899.

Artoria2e5 commented 9 years ago

@MingcongBai So test pending then.

MingcongBai commented 9 years ago

@Arthur2e5

It worked just fine when I made that experimental MAOS build.

Artoria2e5 commented 8 years ago

There are also ways to make debug info smaller: