Bundled LLDB doesn't have support for ZSTD-compressed debugging information enabled

[elsandosgrande]

For example:

warning: (x86_64) /usr/lib64/libSDL2-2.0.so.0 Unable to initialize decompressor for section '.debug_abbrev': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time
warning: (x86_64) /usr/lib64/libSDL2-2.0.so.0 Unable to initialize decompressor for section '.debug_info': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time
warning: (x86_64) /usr/lib64/libc++.so.1 Unable to initialize decompressor for section '.debug_abbrev': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time
warning: (x86_64) /usr/lib64/libc++.so.1 Unable to initialize decompressor for section '.debug_info': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time
warning: (x86_64) /usr/lib64/libc++abi.so.1 Unable to initialize decompressor for section '.debug_abbrev': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time
warning: (x86_64) /usr/lib64/libc++abi.so.1 Unable to initialize decompressor for section '.debug_info': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time
warning: (x86_64) /usr/lib64/libunwind.so.1 Unable to initialize decompressor for section '.debug_abbrev': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time
warning: (x86_64) /usr/lib64/libunwind.so.1 Unable to initialize decompressor for section '.debug_info': LLVM was not built with LLVM_ENABLE_ZSTD or did not find zstd at build time

The debugging information can be decompressed if I use my own copy of LLDB, but it's not a guarantee that I can always have the same version of LLDB locally as is bundled or a version that is fully compatible — CodeLLDB didn't really work with LLDB 18 when it was built against LLDB 17, although it seems to work with it now that it's built against LLDB 19, at least in my limited testing thus far.

Would it be possible to enable ZSTD support for the bundled copy of LLDB? It would be nice so that I could rely on it as a fallback in case the version of LLDB which CodeLLDB is built against ends up falling behind again.

[vadimcn]

In order to maintain pan-Linux compatibility of the binary I'd need to link zstd statically. At the moment CodeLLDB targets Debian Stretch as the minimal target. Does it even have the libzstd-dev package? I don't see it here. I'd suggest using --compress-debug-sections=zlib

[elsandosgrande]

@vadimcn Given https://unix.stackexchange.com/questions/707250/using-debian-stretch-to-build-5-15-x-kernel-with-zstd-compression-fails-with-inc and https://github.com/ionos-cloud/reprepro/issues/20#issuecomment-1217941792, I have a feeling that ZSTD is available on Stretch as well, but I don't have access to any systems that run it to check.

Also, I fear that the online package search is limited to Buster and newer at this point:

Even BASH isn't listed for Stretch: https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=bash

[elsandosgrande]

A friend just spun up a Debian Stretch Docker container and confirmed that libzstd-dev is indeed present:

╭─[user@localhost ~ ]
╰─% docker run -it debian:stretch
root@88d7d04ad03d:/# sed -i 's/deb\.debian/archive.debian/' /etc/apt/sources.list
root@88d7d04ad03d:/# apt update > /dev/null 2>&1
root@88d7d04ad03d:/# apt search zstd
Sorting... Done
Full Text Search... Done
libzstd-dev/oldoldstable 1.1.2-1 amd64
  fast lossless compression algorithm -- development files

libzstd1/oldoldstable 1.1.2-1 amd64
  fast lossless compression algorithm

zstd/oldoldstable 1.1.2-1 amd64
  fast lossless compression algorithm -- CLI tool

root@88d7d04ad03d:/# 

[vadimcn]

That's good, however it also needs to be static, as well as all of its dependencies (not sure what those are), compiled with -fpic, (so it can be linked into a shared library), and, preferably, also exist for arm and arm64 architectures (since I publish CodeLLDB for those). Then there's also OSX and Windows, though I don't know how common zstd-compressed debug info is on these platforms.

[elsandosgrande]

I can't imagine why ZSTD would be any different from other Stretch packages in those regards, but I can look into that later now that I know that doing so is as easy as spinning up a local Docker container. As far as macOS and Windows go, I sadly have no clue 😅.

[brokenbyte]

Putting a stretch container in a k8s cluster running on Pi 4Bs I get the following results

arch linux/arm64/v8:

╭─[user@localhost ~/tmp/kube ]
╰─% kubectl exec -it stretch-deployment-8b647dbd4-6742q -- /bin/bash
root@stretch-deployment-8b647dbd4-6742q:/# sed -i 's/deb\.debian/archive.debian/' /etc/apt/sources.list
root@stretch-deployment-8b647dbd4-6742q:/# apt update > /dev/null 2>&1
root@stretch-deployment-8b647dbd4-6742q:/# apt search zstd
Sorting... Done
Full Text Search... Done
libzstd-dev/oldoldstable 1.1.2-1 arm64
  fast lossless compression algorithm -- development files

libzstd1/oldoldstable 1.1.2-1 arm64
  fast lossless compression algorithm

zstd/oldoldstable 1.1.2-1 arm64
  fast lossless compression algorithm -- CLI tool

arch linux/arm/v7:

╭─[user@localhost ~/tmp/kube ]
╰─% kubectl exec -it stretch-deployment-5c48bfcc6f-x65k5 -- /bin/bash
root@stretch-deployment-5c48bfcc6f-x65k5:/# sed -i 's/deb\.debian/archive.debian/' /etc/apt/sources.list
root@stretch-deployment-5c48bfcc6f-x65k5:/# apt update > /dev/null 2>&1       
root@stretch-deployment-5c48bfcc6f-x65k5:/# apt search zstd   
Sorting... Done
Full Text Search... Done
libzstd-dev/oldoldstable 1.1.2-1 armhf
  fast lossless compression algorithm -- development files

libzstd1/oldoldstable 1.1.2-1 armhf
  fast lossless compression algorithm

zstd/oldoldstable 1.1.2-1 armhf
  fast lossless compression algorithm -- CLI tool

[vadimcn]

Ok, that helps!
However, I still need to know if the objects in libzstd.a were compiled with the -fpic flag. Something like this: https://stackoverflow.com/questions/1340402/how-can-i-tell-with-something-like-objdump-if-an-object-file-has-been-built-wi

[elsandosgrande]

@vadimcn You mentioned earlier that CodeLLDB targets Debian Stretch, but, unless I'm misreading something, the Azure Pipelines definition file indicates that it actually targets (or is built on) Ubuntu Focal Fossa (20.04): https://github.com/vadimcn/codelldb/blob/fe108b53a0f276c10f02d57d85b93991bbb6e30b/azure-pipelines.yml#L28

[vadimcn]

It's built on Focal (host version doesn't really matter), but it targets Stretch.
I think I've figured out where to get the static libzstd, so I'll add zstd compression in the next release. Thanks!

[elsandosgrande]

Awesome!!