Unable to connect to db after containers restarted, could not open file "global/pg_filenode.map"

[siopao]

Fresh install on Unraid community apps today after not using Immich in a while. Installation went fine, I was able to setup multiple users, uploaded 500+ images, create a shared album, etc. After CA docker backup ran and restarted the containers I started hitting the below.

I also got the same error yesterday on my trial installation. It seems like Immich works fine after the initial deployment, registration, etc but when I restart the containers, I hit the wall below.

I am not sure where to begin resolving as it seems blowing it away and reinstalling is likely to work fine for a while, but then stop working after a restart.

I am able to connect to the postgres14 db with the same credentials via adminer, so I get that it's not a credentials issue and presumably a file permissions issue. Appreciate any insight.

Immich container logs:

[Nest] 888  - 06/26/2023, 2:52:08 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (5)...
error: could not open file "global/pg_filenode.map": Permission denied
    at Parser.parseErrorMessage (/app/immich/server/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/app/immich/server/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/app/immich/server/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/app/immich/server/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

postgres14 container logs:

2023-06-26 02:52:17.765 EDT [33] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": Permission denied
2023-06-26 02:52:20.159 EDT [159] FATAL:  could not open file "global/pg_filenode.map": Permission denied

db files are all owned by nobody:users

root@UNRAID:/mnt/user/Scratch/docker/immich/db# ls -l total 92 -rw------- 1 nobody users 3 Jun 25 10:53 PG_VERSION drwx------ 1 nobody users 32 Jun 25 10:53 base/ drwx------ 1 nobody users 574 Jun 26 02:32 global/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_commit_ts/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_dynshmem/ -rw------- 1 nobody users 4821 Jun 25 10:53 pg_hba.conf -rw------- 1 nobody users 1636 Jun 25 10:53 pg_ident.conf drwx------ 1 nobody users 76 Jun 26 02:32 pg_logical/ drwx------ 1 nobody users 28 Jun 25 10:53 pg_multixact/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_notify/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_replslot/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_serial/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_snapshots/ drwx------ 1 nobody users 0 Jun 26 02:16 pg_stat/ drwx------ 1 nobody users 22 Jun 26 02:32 pg_stat_tmp/ drwx------ 1 nobody users 8 Jun 25 10:53 pg_subtrans/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_tblspc/ drwx------ 1 nobody users 0 Jun 25 10:53 pg_twophase/ drwx------ 1 nobody users 124 Jun 25 23:02 pg_wal/ drwx------ 1 nobody users 8 Jun 25 10:53 pg_xact/ -rw------- 1 nobody users 88 Jun 25 10:53 postgresql.auto.conf -rw------- 1 nobody users 28853 Jun 25 10:53 postgresql.conf -rw------- 1 nobody users 36 Jun 26 02:32 postmaster.opts -rw------- 1 nobody users 94 Jun 26 02:32 postmaster.pid `

[martabal]

Hello, it looks like you are using the postgres docker mod which has been deprecated (see #90)

[hydazz]

Hello, it looks like you are using the postgres docker mod which has been deprecated (see #90)

this is an immich error, i believe its the postgres connector.

have you ran fix permissions in unraid recently? youve stuffed up the permissions on your postgres folder

root@docker-external:/mnt/appdata/postgresql12 # ls -la
total 132
drwx------ 19   70 root  4096 Jun 25 15:00 .
drwxr-xr-x 10 root root  4096 Jun 22 07:49 ..
-rw-------  1   70   70     3 Jun  6 06:26 PG_VERSION
drwx------  6   70   70  4096 Jun  6 06:26 base
drwx------  2   70   70  4096 Jun 25 15:00 global
drwx------  2   70   70  4096 Jun  6 06:26 pg_commit_ts
drwx------  2   70   70  4096 Jun  6 06:26 pg_dynshmem
-rw-------  1   70   70  4782 Jun  6 06:26 pg_hba.conf
-rw-------  1   70   70  1636 Jun  6 06:26 pg_ident.conf
drwx------  4   70   70  4096 Jun 26 10:50 pg_logical
drwx------  4   70   70  4096 Jun  6 06:26 pg_multixact
drwx------  2   70   70  4096 Jun 25 15:00 pg_notify
drwx------  2   70   70  4096 Jun  6 06:26 pg_replslot
drwx------  2   70   70  4096 Jun  6 06:26 pg_serial
drwx------  2   70   70  4096 Jun  6 06:26 pg_snapshots
drwx------  2   70   70  4096 Jun 25 15:00 pg_stat
drwx------  2   70   70  4096 Jun 26 10:52 pg_stat_tmp
drwx------  2   70   70  4096 Jun  6 06:26 pg_subtrans
drwx------  2   70   70  4096 Jun  6 06:26 pg_tblspc
drwx------  2   70   70  4096 Jun  6 06:26 pg_twophase
drwx------  3   70   70  4096 Jun 23 04:25 pg_wal
drwx------  2   70   70  4096 Jun  6 06:26 pg_xact
-rw-------  1   70   70    88 Jun  6 06:26 postgresql.auto.conf
-rw-------  1   70   70 26693 Jun  6 06:26 postgresql.conf
-rw-------  1   70   70    24 Jun 25 15:00 postmaster.opts
-rw-------  1   70   70    94 Jun 25 15:00 postmaster.pid

as for

[Nest] 888  - 06/26/2023, 2:52:08 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (5)...
error: could not open file "global/pg_filenode.map": Permission denied
    at Parser.parseErrorMessage (/app/immich/server/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/app/immich/server/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/app/immich/server/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/app/immich/server/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

🤷 @martabal

[siopao]

Thanks both. I wasn't intentionally using the docker mod. I saw this note in the Immich Community App for Unraid, but I ran redis in its own container.

Both postgres and Immich we're built with the default permissions for their respective apps from CA.

If that CA version of Immich isn't up to date, I'll try again using compose when I get to my computer.

[siopao]

Hello, it looks like you are using the postgres docker mod which has been deprecated (see #90)

Is there a layperson explanation about the postgresql docker mod that was deprecated? Looking at those steps to "migrate" to a new postgresql container, it looks like it's effectively repeating what I already did to set up the containers fresh. I setup a postgresql14 container from scratch and referenced that when I completed the Immich setup the first time. So I don't follow what migrating to a new postgresql14 container would accomplish. If I need to figure out how to upgrade to 15 instead, I'll work on that, but I already started with 3 separate containers (redis, postgresql14, and immich), so I don't think migrating/rebuilding from existing postgresql14 container to a new container that is the effectively the same would make a difference.

Edit: As I understand it, the docker mod is basically embedding the other apps (postgresql or redis) within the main Immich container. I didn't do this and the Immich container is referencing the IP, post, credentials of the separate postgresql14 container on the same IP with a bridged network. So based on my understanding, I am not using the docker mod and am instead setup how the linked post suggested.

Regarding file permissions for the db. I didn't specify any unique PUID/GUID values for either the postgresql14 or immich containers. If they don't look right, the only thing I can think of is that I have the db stored is a subfolder of the immich appdata folder. E.g.:

Immich appdata: /mnt/cache/Scratch/docker/immich
Postgresql14 db folder: /mnt/cache/Scratch/docker/immich/db

[martabal]

Ooh, that's what I feared. Since the Immich Docker changes the file permissions in the /config folder, it also affects the permissions of the Postgres folder. Since Immich and Postgres have different permissions, it can cause issues. I'm not sure if this is the root cause of your issue, but it definitely could be. You should set the database folder outside the Immich app data with something like this:

Immich appdata: /mnt/cache/Scratch/docker/immich
Postgresql14 db folder: /mnt/cache/Scratch/docker/postgres

[siopao]

Thanks for the suggestion. I will give that a shot next. This was the same approach I'd taken with many other containers, but using mariadb or other databases, so perhaps they have different permission controls that don't conflict like this.

[siopao]

I'm going to presume this is resolved now and see how it goes for a few days. I started from scratch, set the postgresql15 app data folder outside the immich app data folder and it looks like below by default. It seems to be ok after restarting each container a few times and using the CLI tool for bulk uploads.

That you both for the help and hopefully this does the trick.