search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
9.01k stars 1.08k forks source link

[YSQL] Capture stderr in failed postgres process start #12659

Open tylarb opened 2 years ago

tylarb commented 2 years ago

Jira Link: DB-552

Description

Currently, we only capture the error code on a postgres process start, like:

PostgreSQL server exited with code 256

This is a problem in the case of a postgres process failing to start before it can log to postgres.log because then there's no way to see the failure message, which would be printed to stderr.

In one case, we got lucky and could run the full postgres command on the command line itself...

/home/yugabyte/yb-software/yugabyte-2.12.3.0-b19-centos-x86_64/postgres/bin/postgres -D /mnt/d0/pg_data -p 5433 -h 0.0.0.0 -k /tmp/.yb.8228272234018189749 -c unix_socket_permissions=0700 -c logging_collector=on -c log_directory=/mnt/d0/yb-data/tserver/logs -c yb_pg_metrics.node_name=yb-cloud-c-xroisrzkkjezdcudxrxaj6sff4-n2 -c yb_pg_metrics.port=13000 -c config_file=/mnt/d0/pg_data/ysql_pg.conf -c hba_file=/mnt/d0/pg_data/ysql_hba.conf

Showing us the actual issue...

postgres: superuser_reserved_connections (3) plus max_wal_senders (10) must be less than max_connections (2)

so it would be great if we could capture the stderr of postgres process failures to speed resolution of this type of issue.

foucher commented 6 months ago
  1. This problem occurs for initdb as well.
  2. It may be possible to deal with this by altering class PgWrapper to do an append-redirect (2>>) to <processname>_<pid>.STDERR (pid to differentiate different instantiations of the same process) in the log directory at the time of creating the process, and then tools might transparently pick it up.
foucher commented 2 months ago

Hari added a mechanism to save stderr for some processes run on the yb master in some situations in the pg15-upgrade branch, for example: https://github.com/yugabyte/yugabyte-db/blob/f86bd6cff3215f7b5f8d16865fe79fb5c90ac2c0/src/yb/yql/pgwrapper/pg_wrapper.cc#L872

It's not streaming stderr, but at least it gets collected.

sushantrmishra commented 2 months ago

Also consider K8, systemd deployments to capture PostgreSQL stderr logs.