Segmentation Fault

[MichaelDBA]

Using CentOS Stream release 8 and PostgreSQL 14.2.

Segmentation fault was generated causing PG to restart, after I connected to PG14 and queried the pg_available_extensions view:

You are now connected to database "pgbench" as user "postgres".
pgbench=# select * from pg_available_extensions;
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

Here is the PG log before and after restoring the connection:

2022-04-30 07:35:33 EDT [2408]: [8-1]     0 00000: LOG:  database system is ready to accept connections
2022-04-30 07:44:40 EDT [2408]: [9-1]     0 00000: LOG:  server process (PID 2758) was terminated by signal 11: Segmentation fault
2022-04-30 07:44:40 EDT [2408]: [10-1]     0 00000: DETAIL:  Failed process was running: select * from pg_available_extensions;
2022-04-30 07:44:40 EDT [2408]: [11-1]     0 00000: LOG:  terminating any other active server processes
2022-04-30 07:44:40 EDT [2763]: [1-1] postgres pgbench [unknown] [local] 0 57P03: FATAL:  the database system is in recovery mode
2022-04-30 07:44:40 EDT [2408]: [12-1]     0 00000: LOG:  all server processes terminated; reinitializing
2022-04-30 07:44:40 EDT [2770]: [1-1]     0 00000: LOG:  database system was interrupted; last known up at 2022-04-30 07:35:33 EDT
2022-04-30 07:44:40 EDT [2770]: [2-1]     0 00000: LOG:  database system was not properly shut down; automatic recovery in progress
2022-04-30 07:44:40 EDT [2770]: [3-1]     0 00000: LOG:  redo starts at 2/9C0000A0
2022-04-30 07:44:40 EDT [2770]: [4-1]     0 00000: LOG:  invalid record length at 2/9C0000D8: wanted 24, got 0
2022-04-30 07:44:40 EDT [2770]: [5-1]     0 00000: LOG:  redo done at 2/9C0000A0 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2022-04-30 07:44:41 EDT [2770]: [6-1]     0 00000: LOG:  checkpoint starting: end-of-recovery immediate
2022-04-30 07:44:41 EDT [2770]: [7-1]     0 00000: LOG:  checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.005 s, sync=0.001 s, total=0.109 s; sync files=0, longest=0.000 s, average=0.000 s;
 distance=0 kB, estimate=0 kB
2022-04-30 07:44:41 EDT [2408]: [13-1]     0 00000: LOG:  database system is ready to accept connections

Reconnecting, I was able to query it again without any more errors. Here is the output from the make install command:

gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o createas.o createas.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o matview.o matview.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pg_ivm.o pg_ivm.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -shared -o pg_ivm.so  createas.o matview.o pg_ivm.o -L/usr/pgsql-14/lib  -Wl,--as-needed -L/usr/lib64  -L/usr/lib64 -Wl,--as-needed -Wl,-rpath,'/usr/pgsql-14/lib',--enable-new-dtags
/usr/bin/clang -Wno-ignored-attributes -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -O2  -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -flto=thin -emit-llvm -c -o createas.bc createas.c
In file included from createas.c:44:
./pg_ivm.h:12:9: warning: '_PG_IVM_H_' is used as a header guard here, followed by #define of a different macro [-Wheader-guard]
#ifndef _PG_IVM_H_
        ^~~~~~~~~~
./pg_ivm.h:13:9: note: 'PG_IVM_H_' is defined here; did you mean '_PG_IVM_H_'?
#define PG_IVM_H_
        ^~~~~~~~~
        _PG_IVM_H_
1 warning generated.
/usr/bin/clang -Wno-ignored-attributes -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -O2  -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -flto=thin -emit-llvm -c -o matview.bc matview.c
In file included from matview.c:40:
./pg_ivm.h:12:9: warning: '_PG_IVM_H_' is used as a header guard here, followed by #define of a different macro [-Wheader-guard]
#ifndef _PG_IVM_H_
        ^~~~~~~~~~
./pg_ivm.h:13:9: note: 'PG_IVM_H_' is defined here; did you mean '_PG_IVM_H_'?
#define PG_IVM_H_
        ^~~~~~~~~
        _PG_IVM_H_
1 warning generated.
/usr/bin/clang -Wno-ignored-attributes -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -O2  -I. -I./ -I/usr/pgsql-14/include/server -I/usr/pgsql-14/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -flto=thin -emit-llvm -c -o pg_ivm.bc pg_ivm.c
In file included from pg_ivm.c:30:
./pg_ivm.h:12:9: warning: '_PG_IVM_H_' is used as a header guard here, followed by #define of a different macro [-Wheader-guard]
#ifndef _PG_IVM_H_
        ^~~~~~~~~~
./pg_ivm.h:13:9: note: 'PG_IVM_H_' is defined here; did you mean '_PG_IVM_H_'?
#define PG_IVM_H_
        ^~~~~~~~~
        _PG_IVM_H_
1 warning generated.
/usr/bin/mkdir -p '/usr/pgsql-14/lib'
/usr/bin/mkdir -p '/usr/pgsql-14/share/extension'
/usr/bin/mkdir -p '/usr/pgsql-14/share/extension'
/usr/bin/install -c -m 755  pg_ivm.so '/usr/pgsql-14/lib/pg_ivm.so'
/usr/bin/install -c -m 644 .//pg_ivm.control '/usr/pgsql-14/share/extension/'
/usr/bin/install -c -m 644 .//pg_ivm--1.0.sql  '/usr/pgsql-14/share/extension/'
/usr/bin/mkdir -p '/usr/pgsql-14/lib/bitcode/pg_ivm'
/usr/bin/mkdir -p '/usr/pgsql-14/lib/bitcode'/pg_ivm/
/usr/bin/install -c -m 644 createas.bc '/usr/pgsql-14/lib/bitcode'/pg_ivm/./
/usr/bin/install -c -m 644 matview.bc '/usr/pgsql-14/lib/bitcode'/pg_ivm/./
/usr/bin/install -c -m 644 pg_ivm.bc '/usr/pgsql-14/lib/bitcode'/pg_ivm/./
cd '/usr/pgsql-14/lib/bitcode' && /usr/bin/llvm-lto -thinlto -thinlto-action=thinlink -o pg_ivm.index.bc pg_ivm/createas.bc pg_ivm/matview.bc pg_ivm/pg_ivm.bc

No more bad side-effects occurred after this initial setup problem. I was able to create IMMVs, query them, update underlying tables, and query them again with no problems and saw the expected gains in performance as well.

[yugo-n]

@MichaelDBA Thank you for your feedback!

I fixed the compiler warning about the head guard macro. However, we have not succeeded to reproduce the segmentation fault. Could you please share the procedures to reproduce if you could still observe it? Otherwise we would like to close this issue for now, and we will reopen it when we observe or get a report about this phenomena again.

[MichaelDBA]

Thanks for asking. Yes, please close it since I cannot reproduce it either.