yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.93k stars 1.06k forks source link

[YSQL] ysql_conn_mgr WARNING: missing error text #18908

Open jasonyb opened 1 year ago

jasonyb commented 1 year ago

Jira Link: DB-7763

Description

Starting up a cluster ysql_conn_mgr enabled, it gives warnings right away.

bin/yugabyted start --tserver_flags 'ysql_pg_conf_csv={yb_debug_report_error_stacktrace=true},enable_ysql_conn_mgr=true,allowed_preview_flags_csv=enable_ysql_conn_mgr' --ui false

or

bin/yb-ctl start --tserver_flags '"ysql_pg_conf_csv=yb_debug_report_error_stacktrace=true",enable_ysql_conn_mgr=true,allowed_preview_flags_csv=enable_ysql_conn_mgr'

Postgres logs have multiple of the following

2023-08-29 18:21:38.802 PDT [3469697] WARNING:  missing error text                                                                                                                                                                                                                                                                                                           
2023-08-29 18:21:38.802 PDT [3469697] HINT:  shmkey=1179675                                                                                                                                                                                                                                                                                                                  
  ../../src/yb/yql/pggate/util/ybc_util.cc:338:                                                           @     0x7fd73001f324  YBCGetStackTrace                                                                                                                                                                                                                             
  ../../../../../../../src/postgres/src/backend/utils/error/elog.c:1340:                                  @           0xa45ba7  errhint                                                                                                                                                                                                                                      
  ../../../../../../../src/postgres/src/backend/utils/misc/yb_ysql_conn_mgr_helper.c:610:                 @           0xa7b04e  YbCreateClientId                                                                                                                                                                                                                             
  ../../../../../../src/postgres/src/backend/libpq/auth.c:734:                                            @           0x783030  ClientAuthentication                                                                                                                                                                                                                         
  ../../../../../../src/postgres/src/backend/tcop/postgres.c:5862:                                        @           0x8f26e6  PostgresMain                                                                                                                                                                                                                                 
  ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4658:                                @           0x85163a  BackendRun                                                                                                                                                                                                                                   
  ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4296:                                @           0x85163a  BackendStartup                                                                                                                                                                                                                               
  ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1775:                                @           0x85163a  ServerLoop                                                                                                                                                                                                                                   
  ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1431:                                @           0x853ac2  PostmasterMain                                                                                                                                                                                                                               
  ../../../../../../src/postgres/src/backend/main/main.c:234:                                             @           0x79c925  PostgresServerProcessMain                                                                                                                                                                                                                    
      @           0x79c945                                                                                                                                                                                                                                                                                                                                                   
      @     0x7fd72be3cd84                                                                                                                                                                                                                                                                                                                                                   
      @           0x4a783d                                                                                                                                                                                                                                                                                                                                                   
      @ 0xffffffffffffffff

That stack was from alma8, fastdebug, gcc11, commit abb1f4c98d69a608240b86a72a369ce480b7726e build. Also reproduced on release, clang15.

Warning: Please confirm that this issue does not contain any sensitive information

jasonyb commented 1 year ago

Sometimes, this happens:

yugabyte=# select * from pg_stat_activity;
ERROR:  Error at shmat for shared memory segment with id '1212500'. Invalid argument
../../src/yb/yql/pggate/util/ybc_util.cc:338:                                                           @     0x7fb932fca324  YBCGetStackTrace
../../../../../../../src/postgres/src/backend/utils/error/elog.c:1112:                                  @           0xa44984  errmsg
../../../../../../../src/postgres/src/backend/utils/misc/yb_ysql_conn_mgr_helper.c:78:                  @           0xa79f3c  attach_shmem
../../../../../../../src/postgres/src/backend/utils/misc/yb_ysql_conn_mgr_helper.c:262:                 @           0xa7a285  check_resize_needed
../../../../../../../src/postgres/src/backend/utils/misc/yb_ysql_conn_mgr_helper.c:299:                 @           0xa7a285  resize_shmem_if_needed
../../../../../../../src/postgres/src/backend/utils/misc/yb_ysql_conn_mgr_helper.c:452:                 @           0xa7a285  YbUpdateSharedMemory
../../../../../../../src/postgres/src/backend/access/transam/xact.c:3102:                               @           0x56058c  CommitTransactionCommand
../../../../../../src/postgres/src/backend/tcop/postgres.c:2620:                                        @           0x8e7ed5  finish_xact_command
../../../../../../src/postgres/src/backend/tcop/postgres.c:1242:                                        @           0x8eee7f  exec_simple_query
../../../../../../src/postgres/src/backend/tcop/postgres.c:4753:                                        @           0x8eee7f  yb_exec_simple_query_impl
../../../../../../src/postgres/src/backend/tcop/postgres.c:4721:                                        @           0x8ec2d7  yb_exec_query_wrapper_one_attempt
../../../../../../src/postgres/src/backend/tcop/postgres.c:4745:                                        @           0x8edb15  yb_exec_query_wrapper
../../../../../../src/postgres/src/backend/tcop/postgres.c:4768:                                        @           0x8edb69  yb_exec_simple_query
../../../../../../src/postgres/src/backend/tcop/postgres.c:5393:                                        @           0x8f0548  PostgresMain
../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4658:                                @           0x85163a  BackendRun
../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4296:                                @           0x85163a  BackendStartup
../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1775:                                @           0x85163a  ServerLoop
../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1431:                                @           0x853ac2  PostmasterMain
../../../../../../src/postgres/src/backend/main/main.c:234:                                             @           0x79c925  PostgresServerProcessMain
    @           0x79c945 
    @     0x7fb92ede7d84 
    @           0x4a783d 
    @ 0xffffffffffffffff

And sometimes, within the same session, running it again works fine.