db-postgres: Production build unstable

[yobottehg]

Link to reproduction

No response

Payload Version

3.0-beta.67

Node Version

20 LTS

Next.js Version

15.0.canary.53

Describe the Bug

Our infrastucture provider which hosts our payload docker images and provides the postgres service told us that the CMS service is / was not reachable multiple times.

After scanning all the logs the onliest thing that correlates with the downtime reportings and which in my opinion could cause such downtimes are the following postgres logs:

2024-08-02 13:06:01.274 CEST [1921099] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.274 CEST [1921144] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.274 CEST [1920682] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.274 CEST [1920369] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.278 CEST [1920297] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.278 CEST [1920298] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.278 CEST [1920681] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.280 CEST [1921263] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 13:06:01.282 CEST [1921636] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:16:49.822 CEST [1931024] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:16:49.822 CEST [1931025] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:16:49.822 CEST [1924196] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:16:49.826 CEST [1931022] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:16:49.830 CEST [1924620] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:16:49.830 CEST [1922867] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.204 CEST [1936797] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.204 CEST [1936793] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.204 CEST [1936795] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.204 CEST [1936794] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.208 CEST [1937035] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.209 CEST [1936644] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.209 CEST [1936643] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.209 CEST [1936642] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction
2024-08-02 14:21:32.209 CEST [1936641] sa-cms-db@cms LOG:  unexpected EOF on client connection with an open transaction

any way around that / way forward would be greatly appreciated.

Reproduction Steps

We have a production setup of payload v3 beta 67 in a docker container (using non-standalone build). Postgres version is v16

We have multiple authors using the system concurrently.

We got the 502 / 504 gateways timeout errors reported and these are the onliest things in the logs that in our opinion could cause such downtimes.

Adapters and Plugins

db-postgres

[DanRibbens]

We made improvements to the database adapters in more recent versions including changes to transactions. Try the newest beta v3.0.0-beta.73.

[yobottehg]

@DanRibbens I would love to but I can't because of the mentioned problems in https://github.com/payloadcms/payload/issues/7492

[DanRibbens]

We're stuck without steps to reproduce this issue.

We're working with drizzle on 7492.

I am going to close this since there isn't anything we can at this time. Tag me if you have any new info for us and we can reopen it.

[nunoci]

DanRibbens we have lot of issues on railway I have to restart the database or the nodejs like 5 times or more a day. is the fix on 2.x too?

[DanRibbens]

DanRibbens we have lot of issues on railway I have to restart the database or the nodejs like 5 times or more a day. is the fix on 2.x too?

Hey @nunoci,

I'm sure this is frustrating. I can't speak to Railway in particular as I'm not as familiar. If you want to disable the use of transactions you add do so by setting payload.db.beginTransaction = undefined. You could do set that in your onInit of the payload config.

It could be something you leave off for your environment or use temporarily to troubleshoot. At this point the DB adapter for 3.x has diverged too much to bring it all back to 2.x. I did have a branch to update drizzle-kit and drizzle-orm which we could possibly merge after some more work, but I don't think it will solve this particular problem.

[github-actions[bot]]

This issue has been automatically locked. Please open a new issue if this issue persists with any additional detail.