4.2.0 Add support for uclibc/mulibc/Alpine

[monikamarko]

REL: 4.2.0 Ref: C-3957

[coccyx]

@iapaddler @jrcheli to spike on effort

[iapaddler]

Go Dynamic Execs on ld-musl.so

Go dynamic apps lod and run without apparent issue. However, the pre-loaded library constructor is never called. We can interpose, but we aren't emitting any detail.

The root cause for this behavior is a subtle difference between the ld-musl.so and ld-linux.so (glibc) dynamic linker behavior. glibc seems to call shared object constructors in dynamic linker context (_dl_start_user), while musl libc executes constructors in libc_start_main called by crt1.c. The musl libc dynamic linker loads the application along with preloaded and dependent shared objects. Then it jumps to the application entry point. Typically, the application entry point is code defined in crt1.c, when linked with a musl compatible tool chain. But, in the case when the application has been linked with the internal Go linker, libc_start_main is never invoked. The internal Go linker does not link against crt1.o and just calls the main function directly from the startup code in the executable.

[iapaddler]

Go Dynamic Execs

Given that the library is loaded and preload is possible, an approach could be to interpose a function that the Go runtime calls during it's init and call the library constructor directly.

With experimentation, it can be seen that the Go runtime calls pthread_create() as part of init. Interposing pthread_create() and calling the constructor, ensuring that it's only called once, appears to work. We see Go functions interposed and data is emitted.

As part of library init we get the process name. Both glibc and libmusl use the global variable _program_invocation_shortname. However, since we are early in the init procedure or because lib constructors weren't called, the variable isn't initialized when we need it. So, we drop back to getting the exe path and basename() the string to get the process name.

[iapaddler]

Go Static Execs on Musl

Go static execs segfault when started with ldscope or preloaded.

It turns out that init of the function pointer object g_fn fails. The use of dlsym() with the handle RTLD_NEXT has been the default for all cases. However, the libmusl dlsym() doesn't resolve symbols using RTLD_NEXT when the lib is loaded from dlopen(); the case for Ge static execs. It sort of makes sense in that the handle indicates the use of the next objects symbol table. It's actually not clear in the man page how this is supposed to work. It appears that glibc interprets RTLD_NEXT behavior more broadly than libmusl.

The use of RTLD_DEFAULT works. The symbols for the function pointer object are resolved in the Go static case on libmusl. Added a get handle function to determine which handle to use when populating the functions. Changed several places where we were using internal function pointers to use the common function pointer object.

[iapaddler]

Libmusl Doesn't Support Direct Execution

A glibc shared lib can be executed from the command line. It's a bit of hackery to get this to work. It's a nice option. It's used in libscope to emit help and config messages. Not all loaders support this model. ld-musl.so does not.

Might need to move the help and config output to ldscope.

[iapaddler]

No libresolv in a Musl Distro

We use libresolve to parse DNS answer packets. Looks like this will need to be modified without the use of libresolve.

[iapaddler]

Console I/O and libmusl

On glibc we had to hook an internal function in order to ensure we emitted all console I/O. The same function doesn't exist in libmusl. Depending on the public interfaces is better in libmusl, but it appears that there is missing console detail. Will need to look into libmusl code.

[iapaddler]

• single lib built on ubuntu runs on Alpine
• ldscope extracts ldscopedyn, ldscopedyn extracts libscope.so
• inspect .interp for loader strings to determine glibc vs libmusl
• update the loader string in .interp for glibc exec on libmusl
• create a soft link from glibc loader to musl loader
• create a dir for extraction & sym link using a libscope version string
• set ld lib path env var in the proper case/scenario
• go dynamic, static and cgo exec work and emit expected events
• removal of dependency on libresolv
• DNS events are emitted as expected
• no change to interposed funcs remove for glibc __write
• all help text is handled in ldscopedyn, not in os.c
• no lib main (help text moved)
• dependency on dl_argv removed
• shm_open issues resolved
• all console writes, stdout, stderr, stream and direct I/O, captured
• command line switch added to ldscope to optionally specify a sym link dir
• removed dependency on linking with crt0/crt1
• added interposition of pthread_create to enable Go apps to init libscope.so
• remove dependency on program_invocation_shortname being init'd at constructor time
• ensure that multiple g_fn objects exist in multiple execs
• unit tests all run
• /proc/pid/maps format is different using ld-musl. when running with Go dynamic execs the extraction of the base address needs to change.
• creation of auxv is different when run with ld-musl. the use of an environ global variable is updated in glibc, but not in libmnusl. need to use a calculated address for aux location.
• go_switch segfaults on cgo static execs. turns out that the Go tools or runtime on Alpine do not update a TLS value in an 'm' object. added code to validate each object and fall back to getting the 'fs' value from the kernel if an object can not be validated.
• 6+ additional symbols are included in libscope from the addition of libssl and libcrypto. need to interpose these and call the appropriate code when run with libmusl.
• elf header details change between a dynamic exec and a dynamic shared object. this turns up when updating the loader string on older distros. found this on an Debian 8 distro. we need to reference the .interp section from ph_offset and not ph_v addr.