search

vmware-archive / sre-test

Greenplum - Open Source SRE test project.
2 stars 0 forks source link

debug_assertions parameter issue #174

Closed bala-cg closed 2 years ago

bala-cg commented 3 years ago

Page No : 1320

Issue: changing debug_assertions value to "on", did not start the DB when we try to do a Restart.

Work around: able to start the DB after changing the parameter manually in all segments (Master, stand by Master and all segments).

Master pg_log showed below error.

2021-07-12 15:21:09.028913 UTC,,,p5940,th836245632,,,,0,,,seg-1,,,,,"LOG","22023","assertion checking is not supported by this build",,,,,,,,"call_bool_check_hook","guc.c",9461, 2021-07-12 15:21:09.029182 UTC,,,p5940,th836245632,,,,0,,,seg-1,,,,,"FATAL","F0000","configuration file ""/data1/gpdb/master/gpseg-1/postgresql.conf"" contains errors",,,,,,,,"ProcessConfigFile","guc-file.l",460, [gpadmin@mdw_ipv4 pg_log]$

gpadmin=# gpadmin=# set debug_assertions = 'on'; ERROR: assertion checking is not supported by this build gpadmin=# \q [gpadmin@mdw_ipv4 ~]$ [gpadmin@mdw_ipv4 ~]$ gpconfig -s debug_assertions Values on all segments are consistent GUC : debug_assertions Master value: off Segment value: off [gpadmin@mdw_ipv4 ~]$ gpconfig -c debug_assertions -v on 20210712:15:19:29:005586 gpconfig:mdw_ipv4:gpadmin-[INFO]:-completed successfully with parameters '-c debug_assertions -v on' [gpadmin@mdw_ipv4 ~]$ [gpadmin@mdw_ipv4 ~]$ gpconfig -s debug_assertions Values on all segments are consistent GUC : debug_assertions Master value: off Segment value: off [gpadmin@mdw_ipv4 ~]$ [gpadmin@mdw_ipv4 ~]$

[gpadmin@mdw_ipv4 ~]$ [gpadmin@mdw_ipv4 ~]$ gpstop -arf 20210712:15:20:11:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Starting gpstop with args: -arf 20210712:15:20:11:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Gathering information and validating the environment... 20210712:15:20:11:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20210712:15:20:11:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Obtaining Segment details from master... 20210712:15:20:12:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 6.16.0 build commit:5650be2b79197fed564dca8d734d10f2a76b876c' 20210712:15:20:12:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='fast' 20210712:15:20:12:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Master segment instance directory=/data1/gpdb/master/gpseg-1 20210712:15:20:12:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20210712:15:20:12:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Terminating processes for segment /data1/gpdb/master/gpseg-1 20210712:15:20:12:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Stopping master standby host smdw_ipv4 mode=fast 20210712:15:20:13:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Successfully shutdown standby process on smdw_ipv4 20210712:15:20:13:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown 20210712:15:20:13:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... 20210712:15:20:13:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-0.00% of jobs completed 20210712:15:20:14:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-100.00% of jobs completed 20210712:15:20:14:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... 20210712:15:20:14:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-0.00% of jobs completed 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-100.00% of jobs completed 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:----------------------------------------------------- 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:- Segments stopped successfully = 12 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:- Segments with errors during stop = 0 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:----------------------------------------------------- 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-No leftover gpmmon process found 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Cleaning up leftover shared memory 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[INFO]:-Restarting System... 20210712:15:20:15:005757 gpstop:mdw_ipv4:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 2 Command was: '$GPHOME/bin/gpstart -a -d /data1/gpdb/master/gpseg-1' rc=2, stdout='20210712:15:20:15:005891 gpstart:mdw_ipv4:gpadmin-[INFO]:-Starting gpstart with args: -a -d /data1/gpdb/master/gpseg-1 20210712:15:20:15:005891 gpstart:mdw_ipv4:gpadmin-[INFO]:-Gathering information and validating the environment... 20210712:15:20:15:005891 gpstart:mdw_ipv4:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 6.16.0 build commit:5650be2b79197fed564dca8d734d10f2a76b876c' 20210712:15:20:15:005891 gpstart:mdw_ipv4:gpadmin-[INFO]:-Greenplum Catalog Version: '301908232' 20210712:15:20:15:005891 gpstart:mdw_ipv4:gpadmin-[INFO]:-Starting Master instance in admin mode 20210712:15:20:15:005891 gpstart:mdw_ipv4:gpadmin-[CRITICAL]:-Failed to start Master instance in admin mode 20210712:15:20:15:005891 gpstart:mdw_ipv4:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 1 Command was: 'env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /data1/gpdb/master/gpseg-1 -l /data1/gpdb/master/gpseg-1/pg_log/startup.log -w -t 600 -o " -p 5432 -c gp_role=utility " start' rc=1, stdout='waiting for server to start.... stopped waiting ', stderr='pg_ctl: could not start server Examine the log output. ' ', stderr='' [gpadmin@mdw_ipv4 ~]$ [gpadmin@mdw_ipv4 ~]$ [gpadmin@mdw_ipv4 ~]$ psql psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"? [gpadmin@mdw_ipv4 ~]$

edespino commented 3 years ago

@bala-cg - I believe this is a known limitation of the gpconfig. The ability to add a GUC value that is invalid or inconsistent with the system build.

Can you confirm there is a documented workaround to get a user out of this situation? They

bala-cg commented 3 years ago

@edespino Yes, I was able to start the DB after changing the parameter to "off" manually in all segments.

When we try to set this value from psql it gives a valid error, probably same error while setting it via gpconfig would be helpful to avoid this issue.

gpadmin=# gpadmin=# set debug_assertions = 'on'; ERROR: assertion checking is not supported by this build gpadmin=#

edespino commented 3 years ago

@ashwinstar for GP commercial releases, I'm leaning toward removing references to the debug_assertions parameter from user documentation and possibly hiding it altogether. Thoughts?

ashwinstar commented 3 years ago

yes we can do that. Ideally, this is SHOW ONLY GUC. It's description is ""Shows whether the running server has assertion checks enabled." So, mostly useful only for developers. I don't see harm either to leave it documented as well. Its PRESET_OPTIONS like data_checksums GUC, who value can't be changed at runtime. Only GUC to get the value, not set the value.