search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.81k stars 1.05k forks source link

[Upgrade][Geo-Partition] Cores generated by REFRESH MV after upgrade in geo-partition case #23229

Open agsh-yb opened 1 month ago

agsh-yb commented 1 month ago

Jira Link: DB-12166

Description

Fatals was observed after the upgrade while validating objects before and after upgrade Observed on 2024.1.1.0-b123 Geo-Partition + Upgrade Base version '2024.1.0.0-b129' (db version before upgrade)

It's significantly observed

Environment:

c5.xlarge (4 cores, 8GB RAM)
rf-3
AZs': ap-south-1a, us-east-1a, us-west-2a

Flags:

m_flags = {
            "db_write_buffer_size": 102400,
            "enable_automatic_tablet_splitting": "true",
            "tablet_split_high_phase_shard_count_per_node": 3000,
            "tablet_split_high_phase_size_threshold_bytes": 102400,  # 100 KB
            "tablet_split_low_phase_size_threshold_bytes": 10240,  # 10 KB
            "tablet_split_low_phase_shard_count_per_node": 2000,
            "tablet_split_limit_per_table": 10000,
            "ignore_null_sys_catalog_entries": "true",
            "ysql_enable_read_request_caching": "true",
           "ysql_catalog_preload_additional_tables":"true",
          "ysql_enable_read_request_caching":"true",
          "ysql_enable_packed_row_for_colocated_table":"true",
          "ycql_enable_packed_row":"true",
        }
        t_flags = {
            "enable_automatic_tablet_splitting": "true",
            "yb_num_shards_per_tserver": 1,
            "ysql_num_shards_per_tserver": 1,
            "ysql_enable_read_request_caching": "true",
"ysql_catalog_preload_additional_tables":"true",
          "ysql_enable_read_request_caching":"true",
          "ysql_enable_packed_row_for_colocated_table":"true",
          "ycql_enable_packed_row":"true",
        }

Runtimeconfig
yb.upgrade.enable_rollback_support: true

global value
yb.skip_version_checks: true

Steps:

testupgrade-aws-rf3-upgrade-geo-partition-2024.1.0.0-b129: Start
    (     0.438s) User Login : Success
    (     0.160s) Refresh YB Version : Success
    (    93.487s) Setup Provider : Success
    (     4.152s) Download old release: 2024.1.0.0-b129 : Success
    (    19.737s) Updating Health Check Interval to 60000 sec : Success
    (  1293.534s) Create universe rsak-isd12622-e44dc6da29-20240716-111100 : Success
    (   205.693s) Start sample workloads : Success (CassandraSecondaryIndex)
    (  5736.791s) Create geo-partitioned schema : Success
    (   905.215s) Bulk copy data : Success
    (   298.515s) Validation YSQL objects created : Success
    (   447.107s) Creating YBC backups in S3 : Success
    (    60.034s) Geo Partition Upgrade Errors: : GHI: #18779, #18778
    (   268.094s) Upgrade Software to 2024.1.1.0-b123 : Success
    (   197.752s) Validation YSQL objects created : Success
    (   133.770s) Validation YSQL objects created : Success
    (   113.774s) Validation YSQL objects created : Success
    (   158.733s) Validation YSQL objects created : Success
    (   229.498s) Finalize upgrade for universe : Success
    (    70.274s) Validation YSQL objects created : Success
    (    12.490s) Check for fatals : Success
    (    28.457s) Validation YSQL objects created : >>> Integration Test Failed <<< 
Found at least one core file in 172.151.25.153.

Cores:

Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/postgres]

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing
[New LWP 112272]
[New LWP 112870]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_pggate.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_pggate_util.so]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/../lib/yb-thirdparty/libkeyutils.so.1]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libconsensus_error.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libcdc_service_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libserver_common.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_client.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libtserver_util.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_pggate_flags.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libtablet_error.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libpg_client_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_fs.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libmaster_rpc.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libmaster_util.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libmaster_error.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libtserver_service_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libremote_bootstrap_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libpg_auto_analyze_service_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libtest_echo_service_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_ql_expr.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_dockv.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libmaster_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libconsensus_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libbackup_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libany_yrpc.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libtserver_admin_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libtserver_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libtablet_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libdocdb_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libdockv_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libencryption_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libcdc_consumer_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libxcluster_producer_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyrpc.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_ash.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_common.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libconsensus_metadata_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libfs_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_bfql.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_bfpg.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_common_base.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_bfcommon.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/librpc_introspection_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_common_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/librpc_base_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/librpc_header_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyrpc_base.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libgutil.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libhistogram_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libpb_util_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libversion_info_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/lib/pg_stat_statements.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/lib/../../lib/yb/libyb_pggate_webserver.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/lib/../../lib/yb/libserver_process.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/lib/../../lib/yb/libserver_base_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/lib/yb_pg_metrics.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/lib/pgaudit.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/lib/pg_hint_plan.so]
Core was generated by `postgres: yugabyte mv_upgrade_db2024_1_0_0 10.150.0.237(43032) REFRESH MATERIAL'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f926ae73acf in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f9266973100 (LWP 112272))]

Thread 2 (Thread 0x7f924ae1d700 (LWP 112870)):
#0  0x00007f926b1f87aa in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f926b48e6ab in std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) () from /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/../lib/yb-thirdparty/libc++.so.1
No symbol table info available.
#2  0x00007f926718aad0 in yb::(anonymous namespace)::LongOperationTrackerHelper::Execute() () from /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so
No symbol table info available.
#3  0x00007f92672b75a0 in yb::Thread::SuperviseThread(void*) () from /home/yugabyte/yb-software/yugabyte-2024.1.0.0-b129-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so
No symbol table info available.
#4  0x00007f926b1f21ca in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f926ae5ee73 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7f9266973100 (LWP 112272)):
#0  0x00007f926ae73acf in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f926ae46ea5 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x000055c5ed304f1c in errfinish ()
No symbol table info available.
#3  0x000055c5ed30d1bf in elog_start ()
No symbol table info available.
#4  0x000055c5ed3025af in YbPgInheritsCacheRelCallback ()
No symbol table info available.
#5  0x000055c5ed2d599a in LocalExecuteInvalidationMessage ()
No symbol table info available.
#6  0x000055c5ed2d5fb8 in AtEOXact_Inval ()
No symbol table info available.
#7  0x000055c5eccf119c in AbortTransaction ()
No symbol table info available.
#8  0x000055c5eccf418a in AbortCurrentTransaction ()
No symbol table info available.
#9  0x000055c5ed154447 in PostgresMain ()
No symbol table info available.
#10 0x000055c5ed093170 in BackendRun ()
No symbol table info available.
#11 0x000055c5ed09228d in ServerLoop ()
No symbol table info available.
#12 0x000055c5ed08d396 in PostmasterMain ()
No symbol table info available.
#13 0x000055c5ecf8c13a in PostgresServerProcessMain ()
No symbol table info available.
#14 0x000055c5ecc3e652 in main ()
No symbol table info available.

cc: @kripasreenivasan

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

agsh-yb commented 1 month ago

Similar cores observed with TestUpgrade.testupgrade-aws-rf3-upgrade-geo-partition-2.21.0.0-b545 base version 2.21.0.0-b545

Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/postgres]

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing
[New LWP 111816]
[New LWP 112359]
[New LWP 111823]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_pggate.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_pggate_util.so]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/../lib/yb-thirdparty/libkeyutils.so.1]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libconsensus_error.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_client.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libtserver_util.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_pggate_flags.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libtablet_error.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libpg_client_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libmaster_rpc.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libmaster_util.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libmaster_error.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libserver_process.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libserver_base_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libserver_common.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_fs.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libtserver_service_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libremote_bootstrap_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libpg_auto_analyze_service_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libtest_echo_service_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_ql_expr.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_dockv.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libmaster_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libconsensus_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libbackup_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libany_yrpc.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libtserver_admin_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libtserver_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libtablet_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libdocdb_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libdockv_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libopid_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libencryption_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libcdc_consumer_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libxcluster_producer_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyrpc.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_ash.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_common.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libconsensus_metadata_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libfs_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_bfql.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_bfpg.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_common_base.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_bfcommon.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/librpc_introspection_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_common_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/librpc_base_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/librpc_header_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyrpc_base.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libgutil.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libhistogram_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libpb_util_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libversion_info_proto.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/lib/pg_stat_statements.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/lib/../../lib/yb/libyb_pggate_webserver.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/lib/yb_pg_metrics.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/lib/pgaudit.so]
Dwarf Error: Cannot handle DW_FORM_strx1 in DWARF reader [in module /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/lib/pg_hint_plan.so]
Core was generated by `postgres: yugabyte mv_upgrade_db2_21_0_0 10.150.0.248(50880) REFRESH MATERIALIZ'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f04a2e9eacf in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f049eac4100 (LWP 111816))]

Thread 3 (Thread 0x7f0484871700 (LWP 111823)):
#0  0x00007f04a3223848 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f049f400815 in yb::ThreadJoiner::Join() () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so
No symbol table info available.
#2  0x00007f049f96d25c in yb::rpc::Reactor::Join() () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyrpc.so
No symbol table info available.
#3  0x00007f049f94a418 in yb::rpc::Messenger::Shutdown() () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyrpc.so
No symbol table info available.
#4  0x00007f04a36b38a2 in void ev::base<ev_async, ev::async>::method_thunk<yb::pggate::PgApiImpl::Interrupter, &yb::pggate::PgApiImpl::Interrupter::AsyncHandler>(ev_loop*, ev_async*, int) () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_pggate.so
No symbol table info available.
#5  0x00007f049f085c3b in ev_invoke_pending () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/../lib/yb-thirdparty/libev.so.4
No symbol table info available.
#6  0x00007f049f089727 in ev_run () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/../lib/yb-thirdparty/libev.so.4
No symbol table info available.
#7  0x00007f049f4031fe in yb::Thread::SuperviseThread(void*) () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so
No symbol table info available.
#8  0x00007f04a321d1ca in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007f04a2e89e73 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f0484070700 (LWP 112359)):
#0  0x00007f04a322345c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f04a34b9d92 in std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/../lib/yb-thirdparty/libc++.so.1
No symbol table info available.
#2  0x00007f049f2dcc9d in yb::(anonymous namespace)::LongOperationTrackerHelper::Execute() () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so
No symbol table info available.
#3  0x00007f049f4031fe in yb::Thread::SuperviseThread(void*) () from /home/yugabyte/yb-software/yugabyte-2.21.0.0-b545-centos-x86_64/postgres/bin/../../lib/yb/libyb_util.so
No symbol table info available.
#4  0x00007f04a321d1ca in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f04a2e89e73 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7f049eac4100 (LWP 111816)):
#0  0x00007f04a2e9eacf in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f04a2e71ea5 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000564cb0eda58c in errfinish ()
No symbol table info available.
#3  0x0000564cb0ee296f in elog_start ()
No symbol table info available.
#4  0x0000564cb08d2b48 in AbortTransaction ()
No symbol table info available.
#5  0x0000564cb08d5f3a in AbortCurrentTransaction ()
No symbol table info available.
#6  0x0000564cb0d29a62 in PostgresMain ()
No symbol table info available.
#7  0x0000564cb0c6b990 in BackendRun ()
No symbol table info available.
#8  0x0000564cb0c6ab6f in ServerLoop ()
No symbol table info available.
#9  0x0000564cb0c65cd6 in PostmasterMain ()
No symbol table info available.
#10 0x0000564cb0b67e8a in PostgresServerProcessMain ()
No symbol table info available.
#11 0x0000564cb081f7d2 in main ()
No symbol table info available.