dongwooklee96 commented 2 years ago

Hi, I tried to analyze psql using uftrace, but I encountered the following error

WARN: Segmentation fault: address not mapped (addr: (nil))
WARN:  if this happens only with uftrace, please consider -e/--estimate-return option.

WARN: Backtrace from uftrace  ( x86_64 dwarf python luajit tui perf sched dynamic )
WARN: =====================================
WARN: [1] (pg_logging_init[55a2f5cea67b] <= main[55a2f5cc2e4f])
WARN: [0] (main[55a2f5cc2e33] <= __libc_start_main[7f0d792dd083])

Please report this bug to https://github.com/namhyung/uftrace/issues.

WARN: child terminated by signal: 11: Segmentation fault

https://user-images.githubusercontent.com/14002238/178113098-a7fbdbf9-b36f-496a-83fb-f79cabaf021f.mov

do you know how to solve it? I use Ubuntu 20.04.4 LTS and Postgres 15beta1

how to install postgres

ref : 9 Steps to Install and Configure PostgreSQL from Source on Linux

clone source code

git clone https://github.com/postgres/postgres.git

configure & make below is build script use ./script
```
#!/bin/bash
```

BRANCH="$1"

if [ -z "$BRANCH" ]; then echo "Usage: $0 " exit 1 fi

change path to your source code!!

SOURCE="/home/postgres/git/pg/dev/$BRANCH" BUILD="/home/postgres/git/pg/dev/build/$BRANCH" INSTALL="/home/postgres/git/pg/dev/install/$BRANCH"

if [ -d "$BUILD" ]; then rm -r "$BUILD" fi

mkdir -p "$BUILD"

if [ -d "$INSTALL" ]; then rm -r "$INSTALL"/* else mkdir "$INSTALL" fi

(cd "$BUILD" && \ CFLAGS="-pg -I/usr/include/mit-krb5 -I/usr/include/python3.8/ -fuse-ld=gold -ggdb -Og -g3 -fno-omit-frame-pointer -U HAVE_POSIX_FADVISE" LDFLAGS=-L/usr/lib/x86_64-linux-gnu/mit-krb5 "$SOURCE"/configure --silent --prefix="$INSTALL" --with-openssl --with-tcl --with-tclconfig=/usr/lib/tcl8.6 --with-perl --w ith-libxml --with-libxslt --with-python --with-gssapi --with-systemd --with-ldap --enable-nls --enable -debug --enable-cassert --enable-tap-tests --enable-depend && \

CFLAGS="-I/usr/include/mit-krb5 -fuse-ld=gold -ggdb -Og -g3 -fno-omit-frame-pointer -U HAVE_POSIX_F

ADVISE" LDFLAGS=-L/usr/lib/x86_64-linux-gnu/mit-krb5 "$SOURCE"/configure --silent --prefix="$INSTALL" --with-ssl=nss --with-tcl --with-tclconfig=/usr/lib/tcl8.6 --with-perl --with-libxml --with-libxslt --with-python --with-gssapi --with-systemd --with-ldap --enable-nls --enable-debug --enable-cassert --enable-tap-tests --enable-depend && \ make -s -j4 PG_TEST_EXTRA='kerberos ssl' && \ make -s -j4 install PG_TEST_EXTRA='kerberos ssl' && \ make -s -j4 check PG_TEST_EXTRA='kerberos ssl' && \ make -s -j4 world PG_TEST_EXTRA='kerberos ssl' && \ make -s -j4 install-world PG_TEST_EXTRA='kerberos ssl' && \ make -s -j4 check-world PG_TEST_EXTRA='kerberos ssl' -O PROVE_FLAGS=-j4 \ )

echo echo "Branch: $BRANCH run completed."


step 3.

cd mkdir data cd bin ./initdb -D ../data ./postmaster -D ../data ./psql uftrace record ./psql

honggyukim commented 2 years ago

Thanks @dongwooklee96. It's confirmed that it doesn't show the bug with --no-libcall. We may have to think we might remove library call tracing from default option.

honggyukim commented 2 years ago

I have tried to build and install as follows.

$ git clone https://github.com/postgres/postgres.git
$ cd postgres
$ ./configure --prefix=`pwd`/build
$ make -j32 install

Then tried to run it as follows, but shows an error.

$ cd build/bin/
$ ./initdb -D ../data
$ ./postmaster -D ../data
$ ./psql
psql: error: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

So I couldn't investigate more in my environment.

JSYoo5B commented 2 years ago

I also discovered similar issue with this. (not postgresql, but SEGFAULT on uftrace recording) I've tested uftrace on Asahi linux(the Apple M1-ported version linux).

Tried foobar example in the uftrace.github.io/slide/

void bar() {

}

void foo() {
  bar();
}

int main() {
  foo();
  bar();
}

$ gcc -pg -g -o foobar foobar.c
$ uftrace ./foobar
WARN: child terminated by signal: 11: Segmentation fault
WARN: cannot open record data: /tmp/uftrace-live-nMiZjE: No data available

$ uftrace --no-libcall ./foobar
# DURATION     TID     FUNCTION
            [1069565] | main() {
            [1069565] |   foo() {
   0.084 us [1069565] |     bar();
   0.875 us [1069565] |   } /* foo */
   0.041 us [1069565] |   bar();
   1.250 us [1069565] | } /* main */

I tried to figure out the reason of SEGFAULT, but it also resolved by adding --no-libcall option on recording.

Testing environment information

Device: Apple MacBook Air M1, 8GB ram, 512GB ssd
Linux Distribution: Asahi linux minimal (https://asahilinux.org/2022/03/asahi-linux-alpha-release/)
Kernel version(uname -a): Linux JSYoo5B-Sombre 5.17.0-rc7-asahi-next-20220310-5-2-ARCH #2 SMP PREEMPT Tue, 22 Mar 2022 14:28:10 +0000 aarch64 GNU/Linux

Other related version infos:


$ ldd foobar
    linux-vdso.so.1 (0x0000ffff04bf8000)
    libc.so.6 => /usr/lib/libc.so.6 (0x0000ffff04a10000)
    /lib/ld-linux-aarch64.so.1 => /usr/lib/ld-linux-aarch64.so.1 (0x0000aaaaf3430000)

$ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-unknown-linux-gnu/12.1.0/lto-wrapper Target: aarch64-unknown-linux-gnu Configured with: /build/gcc/src/gcc/configure --enable-languages=c,c++,fortran,go,lto,objc,obj-c++ --enable-bootstrap --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://github.com/archlinuxarm/PKGBUILDs/issues --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-multilib --disable-werror --host=aarch64-unknown-linux-gnu --build=aarch64-unknown-linux-gnu --with-arch=armv8-a --enable-fix-cortex-a53-835769 --enable-fix-cortex-a53-843419 Thread model: posix Supported LTO compression algorithms: zlib zstd gcc version 12.1.0 (GCC)

$ /usr/lib/libc.so.6 GNU C Library (GNU libc) stable release version 2.35. Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 12.1.0. libc ABIs: UNIQUE ABSOLUTE For bug reporting instructions, please see: https://github.com/archlinuxarm/PKGBUILDs/issues.

JSYoo5B commented 2 years ago

@honggyukim, Maybe you can reference these steps.

build

This step is same as your build instruction

$ git clone https://github.com/postgres/postgres.git
$ cd postgres
$ ./configure --prefix=`pwd`/build
$ make -j32 install

run

Prepare initial DB

$ cd build/bin/
$ ./initdb -D ../data

Run postgres server (master)

$ cd build/bin/
$ ./postmaster -D ../data

Run postgres client

$ cd build/bin/

# need to run with `postgres` user
$ ./psql

# when it fails with error such as `database "username" does not exist`
# (when you want to run without `postgres` user)
$ ./psql --dbname=postgres
psql (16devel)
Type "help" for help.

postgres=#

JSYoo5B commented 2 years ago

@dongwooklee96, I tried build & test postgres, and it seems segment fault not occurs on dynamic tracing. I failed to build postgres with -pg or -finstrument-functions. Is there any easier way to add one of these options by make arguments? (ex. make CFLAGS=-pg)

Following step shows how I tried uftrace record

$ uftrace -V
uftrace v0.11-96-g2bbc ( x86_64 dwarf python luajit tui perf sched dynamic )

$ uftrace record -P=. ./psql --dbname=postgres
psql (16devel)
Type "help" for help.

postgres=#
\q

$ ls uftrace.data
162629.dat                    libgcc_s.so.1.sym          libp11-kit.so.0.3.0.sym
default.opts                  libgmp.so.10.4.1.sym       libpq.so.5.16.sym
info                          libgnutls.so.30.31.0.sym   libpsl.so.5.3.2.sym
ld-linux-x86-64.so.2.sym      libgssapi_krb5.so.2.2.sym  libreadline.so.8.1.sym
libbrotlicommon.so.1.0.9.sym  libhogweed.so.6.4.sym      libresolv.so.2.sym
libbrotlidec.so.1.0.9.sym     libidn2.so.0.3.7.sym       librtmp.so.1.sym
libbz2.so.1.0.4.sym           libk5crypto.so.3.1.sym     libsasl2.so.2.0.25.sym
libcapstone.so.4.sym          libkeyutils.so.1.9.sym     libssh.so.4.8.7.sym
libcom_err.so.2.1.sym         libkrb5.so.3.3.sym         libtasn1.so.6.6.2.sym
libcrypto.so.3.sym            libkrb5support.so.0.1.sym  libtinfo.so.6.3.sym
libc.so.6.sym                 liblber-2.5.so.0.1.7.sym   libunistring.so.2.2.0.sym
libcurl-gnutls.so.4.7.0.sym   libldap-2.5.so.0.1.7.sym   libz.so.1.2.11.sym
libdebuginfod-0.186.so.sym    liblzma.so.5.2.5.sym       libzstd.so.1.4.8.sym
libdw-0.186.so.sym            libm.so.6.sym              psql.sym
libelf-0.186.so.sym           libnettle.so.8.4.sym       sid-8a9f3ed57ad53d99.map
libffi.so.8.1.0.sym           libnghttp2.so.14.20.1.sym  task.txt

namhyung / uftrace

Segmentation fault with postgres #1470

how to install postgres

change path to your source code!!

CFLAGS="-I/usr/include/mit-krb5 -fuse-ld=gold -ggdb -Og -g3 -fno-omit-frame-pointer -U HAVE_POSIX_F

build

run