search

TritonDataCenter / pg_prefaulter

Faults pages into PostgreSQL shared_buffers or filesystem caches in advance of WAL apply
Apache License 2.0
56 stars 13 forks source link

panic: bad, open vs close count not the same after purge #45

Open jclulow opened 6 years ago

jclulow commented 6 years ago

During a recent incident, the prefaulter panicked on more than 200 PostgreSQL instances, with the following set of log messages:

19:01:00.598Z ERROR pg_prefaulter: unable to find WAL files (next step=retrying)
    error: unable to query PostgreSQL checkpoint information: write unix ->/tmp/.s.PGSQL.5432: write: broken pipe: (retriable: true, purge cache: true
19:01:01.599Z LVL70 pg_prefaulter: bad, open vs close count not the same after purge (close-count=769548809, open-count=981315695)
panic: bad, open vs close count not the same after purge

Core files for the failures are tagged with this ticket in thoth.

jclulow commented 6 years ago 
$ thoth ls mtime=30d ticket=joyent/pg_prefaulter#45
NAME             TYPE  TIME                NODE/CMD         TICKET                 
dc84346d3ac1a189 core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
38b045311eced13e core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
3c65fc3bff3325a0 core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
99c0765f19a5787c core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
24989b6f33c6a6ee core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
772e1a041f2920b9 core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
de7fe333d3abe005 core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
e14bc25d30c1c836 core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
230c662c3d7e5f9e core  2018-03-27T19:01:02 pg_prefaulter    joyent/pg_prefaulter#45
a75e4a4805dd7883 core  2018-03-27T19:01:03 pg_prefaulter    joyent/pg_prefaulter#45
cfc9c72cf4b7a0da core  2018-03-27T19:01:03 pg_prefaulter    joyent/pg_prefaulter#45
4cba3db5118197e1 core  2018-03-27T19:01:03 pg_prefaulter    joyent/pg_prefaulter#45
9773837217dc1d8e core  2018-03-27T19:01:03 pg_prefaulter    joyent/pg_prefaulter#45
04926ca5080e3086 core  2018-03-27T19:01:03 pg_prefaulter    joyent/pg_prefaulter#45
...