Cluster down due to "pg_stat_tmp/global.tmp" files not accessing

[ns-berumalu]

Hi

Suddenly we have seen below logs and db went down. Please let us know what was the issue. We are using agens 2.12. 2022-10-18 10:42:23 UTC [2027]: [94-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:23 UTC [2027]: [95-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:24 UTC [2027]: [96-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:24 UTC [2027]: [97-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:24 UTC [402766]: [1-1] 634e8310.6254e 0 postgres pgpool [unknown] 10.240.8.66FATAL: could not open file "global/pg_filenode.map": No such file or directory 2022-10-18 10:42:24 UTC [402767]: [1-1] 634e8310.6254f 0 postgres pgpool [unknown] 10.240.8.66FATAL: could not open file "global/pg_filenode.map": No such file or directory 2022-10-18 10:42:24 UTC [2027]: [98-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:24 UTC [2027]: [99-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [100-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [101-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [102-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [103-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [104-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [105-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [106-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:25 UTC [2027]: [107-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:26 UTC [2027]: [108-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:26 UTC [2027]: [109-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:26 UTC [2027]: [110-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:26 UTC [2027]: [111-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:26 UTC [2027]: [112-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:26 UTC [2027]: [113-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:27 UTC [2027]: [114-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:27 UTC [2027]: [115-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:27 UTC [2027]: [116-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:27 UTC [2027]: [117-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:27 UTC [2027]: [118-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:27 UTC [2027]: [119-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2027]: [120-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2027]: [121-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2027]: [122-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2027]: [123-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2025]: [2-1] 6332764c.7e9 0 LOG: using stale statistics instead of current ones because stats collector is not responding 2022-10-18 10:42:28 UTC [2027]: [124-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2027]: [125-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2027]: [126-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:28 UTC [2027]: [127-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:29 UTC [2027]: [128-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:29 UTC [2027]: [129-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:29 UTC [2027]: [130-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:29 UTC [2027]: [131-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat_tmp/global.tmp": No such file or directory 2022-10-18 10:42:30 UTC [2017]: [8-1] 6332764c.7e1 0 LOG: could not open file "postmaster.pid": No such file or directory 2022-10-18 10:42:30 UTC [2017]: [9-1] 6332764c.7e1 0 LOG: performing immediate shutdown because data directory lock file is invalid 2022-10-18 10:42:30 UTC [2017]: [10-1] 6332764c.7e1 0 LOG: received immediate shutdown request 2022-10-18 10:42:30 UTC [2017]: [11-1] 6332764c.7e1 0 LOG: could not open file "postmaster.pid": No such file or directory 2022-10-18 10:42:30 UTC [2027]: [132-1] 6332764c.7eb 0 LOG: could not open temporary statistics file "pg_stat/global.tmp": No such file or directory 2022-10-18 10:42:30 UTC [2017]: [12-1] 6332764c.7e1 0 LOG: archiver process (PID 2026) exited with exit code 2 2022-10-18 10:42:30 UTC [402774]: [1-1] 634e8316.62556 0 postgres pgpool [unknown] 10.240.8.67FATAL: the database system is shutting down 2022-10-18 10:42:30 UTC [402775]: [1-1] 634e8316.62557 0 postgres pgpool [unknown] 10.240.8.67FATAL: the database system is shutting down 2022-10-18 10:42:30 UTC [2017]: [13-1] 6332764c.7e1 0 LOG: database system is shut down

[emotionbug]

Are you using hotfix version? and, which OS are you using?

[ns-berumalu]

Yes, we are using hotfix binaries on os Ubuntu 20.04.4 LTS