Closed denkyem closed 4 months ago
I tought the same, but just waited a little longer and it worked
I tought the same, but just waited a little longer and it worked
When the container starts, it checks if the folder mounted to /photos
has the correct permissions. If not, it tries to change it. This may take more or less time depending on the size of your library
I tought the same, but just waited a little longer and it worked
Well, that seams about right. After a really long time, it starts
Thanks
I tought the same, but just waited a little longer and it worked
Well, that seams about right. After a really long time, it starts
Thanks
Can you open a new issue if you encounter this problem again, please? This shouldn't happen every time
Are you you guys using GPU acceleration with cuda?
I have the same issue, but interestingly it used to be waaaaay faster (1-2 mins) now its been going for an hour (drives io is heavily used so I assume its trying to change perms - BTW can this be disabled? Or can there be some indicator something is in progress?. Going from v1.107.1-ig297 to v1.107.2-ig302.
Did you remove the cuda
folder in /config/machine-learning
?
not using cuda, just the standard image ghcr.io/imagegenius/immich:v1.107.2-ig302. No cuda folder. Plugged in the v1.108.0-ig303 release. Been waiting on v1.107.2-ig302 for over an hour, will prolly leave it for the night. Still, I'd rather not have it start up for hours each time :/
Can you run these 2 commands inside the Immich container and post the outputs?
find /config -type f | wc -l
time lsiown -R abc:abc /config
root@05cf99ba49f8:/# find /config -type f | wc -l
179
root@05cf99ba49f8:/# time lsiown -R abc:abc /config
real 0m0.015s
user 0m0.000s
sys 0m0.004s
Compose service:
immich:
image: ghcr.io/imagegenius/immich:v1.108.0-ig303
container_name: immich
dns:
- ${PIHOLE_ADDR}
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/Warsaw
- DB_HOSTNAME=immich-postgres
- DB_USERNAME=postgres
- DB_PASSWORD=REDACTED
- DB_DATABASE_NAME=immich
- REDIS_HOSTNAME=immich-redis
- DISABLE_MACHINE_LEARNING=false
- DISABLE_TYPESENSE=false
- DB_PORT=5432
- REDIS_PORT=6379
volumes:
- ${CONFIGS}/immich/config:/config
- ${PHOTOS}/encoded-video:/photos/encoded-video
- ${PHOTOS}/library:/photos/library
- ${PHOTOS}/profile:/photos/profile
- ${PHOTOS}/upload:/photos/upload
- ${PHOTOS_THUMBNAILS}:/photos/thumbs
- ${CONFIGS}/immich/ML:/config/machine-learning #optional
expose:
- 8080
restart: always
devices:
- /dev/dri:/dev/dri
As I mentioned above, 1.107.1 booted up fine. PHOTOS is on nfs
Mmmh, someone had the same issue as you last week but his issue was with his /config
which is not the case for you.
time find /photos -maxdepth 0 \( ! -user abc -o ! -group abc \) -exec lsiown -R abc:abc {} \;
time find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
BTW can this be disabled?
We thought about it, but it's supposed to change the permissions of /photos
only if they are incorrect
root@05cf99ba49f8:/# time find /photos -maxdepth 0 \( ! -user abc -o ! -group abc \) -exec lsiown -R abc:abc {} \;
real 0m0.001s
user 0m0.001s
sys 0m0.000s
root@05cf99ba49f8:/# time find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
real 0m7.863s
user 0m0.000s
sys 0m0.175s
I guess with a library closing in on 1 milion pics even checking can be long? Especially on spinning rust.
sd* are the nfs drives, so smth is happening
Just to be sure, are they network drives ?
Yup, OMV NFS share on a vm on proxmox, next to docker vm also on proxmox where immich resides. Thumbnails are separated to the vm storage on nvme.
root@05cf99ba49f8:/# time find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
real 0m0.017s
user 0m0.004s
sys 0m0.013s
Ok, so you need to change the PUID
and PGID
to have the same values as the user who mounts you network shares.
So root then XD cause mounted via /etc/fstab
Didn't seem to help though, no coming back now I guess.
Will let it play overnight I guess
@martabal ummm so its still taking forever after container recreation. But now on v1.108.0-ig303 the webui is a blank page, at the same time mobile app works lol... (it started after 2hrs) New errors in logs:
[Nest] 208 - 07/11/2024, 7:08:57 AM ERROR [Microservices:MetadataService] Unable to initialize reverse geocoding: Error: ENOENT: no such file or directory, open '/build/geodata/geodata-date.txt'
[Nest] 189 - 07/11/2024, 7:14:04 AM WARN [Api:DuplicateService~6z5e5riw] Unknown keys found: {
"job": {
"objectTagging": {
"concurrency": 50
},
"recognizeFaces": {
"concurrency": 50
}
},
"machineLearning": {
"classification": {
"enabled": true
}
}
}
[Nest] 189 - 07/11/2024, 7:14:04 AM WARN [Api:DuplicateService~me1s9cyq] Unknown keys found: {
"job": {
"objectTagging": {
"concurrency": 50
},
"recognizeFaces": {
"concurrency": 50
}
},
"machineLearning": {
"classification": {
"enabled": true
}
}
}
Also, coming back to v1.107.1-ig297 everything is fine (it even starts faster than before). (webui and app work) but the new errors persist and there's another one:
[Nest] 169 - 07/11/2024, 7:32:50 AM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://127.0.0.1:3003" failed with Error: connect ECONNREFUSED 127.0.0.1:3003
[Nest] 169 - 07/11/2024, 7:32:50 AM ERROR [Microservices:JobService] Error: Machine learning request to "http://127.0.0.1:3003" failed with Error: connect ECONNREFUSED 127.0.0.1:3003
at /app/immich/server/dist/repositories/machine-learning.repository.js:19:19
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async MachineLearningRepository.predict (/app/immich/server/dist/repositories/machine-learning.repository.js:18:21)
at async MachineLearningRepository.detectFaces (/app/immich/server/dist/repositories/machine-learning.repository.js:33:26)
at async PersonService.handleDetectFaces (/app/immich/server/dist/services/person.service.js:275:52)
at async /app/immich/server/dist/services/job.service.js:148:36
at async Worker.processJob (/app/immich/server/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
at async Worker.retryIfFailed (/app/immich/server/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 169 - 07/11/2024, 7:32:50 AM ERROR [Microservices:JobService] Object:
{
"id": "fd549e7d-4464-4ba9-a6e2-297309bae1f4",
"source": "upload"
}
To sum up I changed PGID and PUID to 0 (as its root who mounts nfs) and everything seems to be correctly owned by root user.
Can you use the latest image?
I did use v1.108.0-ig303 and after leaving it overnight to do chown the webui got blank and after container recreation the long start came up again.
Basically any image newer than v1.107.1-ig297 takes forever to start, while v1.107.1-ig297 is fast and works. Also, I really appreciate You taking the time to help here ;)
The latest image is v1.108.0-ig304
and solves the issue with the blank screen
Testing rn
It's going for chown again judging from spinning rust sounds from under my desk. Why is v1.107.1-ig297 not doing chown (or doing it differently) and all later does? Weird
It's going for chown again judging from spinning rust sounds from under my desk. Why is v1.107.1-ig297 not doing chown (or doing it differently) and all later does? Weird
can you add another variable called S6_VERBOSITY
and set it to 2
and post those logs
On v1.108.0-ig304:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service init-migrations: starting
s6-rc: info: service init-envfile: starting
s6-rc: info: service init-config-immich: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
[migrations] started
[migrations] no migrations found
s6-rc: info: service init-migrations successfully started
s6-rc: info: service init-adduser: starting
s6-rc: info: service init-envfile successfully started
s6-rc: info: service legacy-cont-init successfully started
╔═══════════════════════════════╗
__ _____ _____ __
/ / |_ _/ ____| / /
/ / | || | __ / /
/ / | || | |_ | / /
/ / _| || |__| | / /
/_/ |_____\_____| /_/
Baseimage from linuxserver.io
╠═══════════════════════════════╣
To support this applications developer(s) visit:
Immich: https://immich.app/docs/overview/support-the-project
╠═══════════════════════════════╣
User/Group ID:
User UID: 0
User GID: 0
╚═══════════════════════════════╝
s6-rc: info: service init-adduser successfully started
s6-rc: info: service init-os-end: starting
s6-rc: info: service init-os-end successfully started
s6-rc: info: service init-crontab-config: starting
s6-rc: info: service init-config: starting
s6-rc: info: service init-config successfully started
s6-rc: info: service init-check-variables: starting
s6-rc: info: service init-check-variables successfully started
s6-rc: info: service init-crontab-config successfully started
And this is on v1.107.1-ig297 where it starts up ok:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service init-migrations: starting
s6-rc: info: service init-envfile: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
[migrations] started
[migrations] no migrations found
s6-rc: info: service init-migrations successfully started
s6-rc: info: service init-adduser: starting
s6-rc: info: service init-envfile successfully started
s6-rc: info: service legacy-cont-init successfully started
╔═══════════════════════════════╗
__ _____ _____ __
/ / |_ _/ ____| / /
/ / | || | __ / /
/ / | || | |_ | / /
/ / _| || |__| | / /
/_/ |_____\_____| /_/
Baseimage from linuxserver.io
╠═══════════════════════════════╣
To support this applications developer(s) visit:
Immich: https://immich.app/docs/overview/support-the-project
╠═══════════════════════════════╣
User/Group ID:
User UID: 0
User GID: 0
╚═══════════════════════════════╝
s6-rc: info: service init-adduser successfully started
s6-rc: info: service init-os-end: starting
s6-rc: info: service init-os-end successfully started
s6-rc: info: service init-config: starting
s6-rc: info: service init-config successfully started
s6-rc: info: service init-check-variables: starting
s6-rc: info: service init-check-variables successfully started
s6-rc: info: service init-gpu-acceleration: starting
+ config_path=/config/machine-learning
+ '[' '' = cuda ']'
s6-rc: info: service init-gpu-acceleration successfully started
s6-rc: info: service init-config-immich: starting
s6-rc: info: service init-config-immich successfully started
s6-rc: info: service init-video-immich: starting
s6-rc: info: service init-test-run: starting
s6-rc: info: service init-test-run successfully started
s6-rc: info: service init-config-end: starting
s6-rc: info: service init-config-end successfully started
s6-rc: info: service init-mods: starting
s6-rc: info: service init-mods successfully started
s6-rc: info: service init-mods-package-install: starting
**** permissions for /dev/dri/renderD128 are good ****
s6-rc: info: service init-mods-package-install successfully started
s6-rc: info: service init-mods-end: starting
s6-rc: info: service init-mods-end successfully started
s6-rc: info: service init-custom-files: starting
**** permissions for /dev/dri/card1 are good ****
[custom-init] No custom files found, skipping...
s6-rc: info: service init-custom-files successfully started
**** permissions for /dev/dri/card0 are good ****
s6-rc: info: service init-services: starting
s6-rc: info: service init-video-immich successfully started
s6-rc: info: service init-services successfully started
s6-rc: info: service svc-server: starting
Detected CPU Cores: 16
Starting api worker
Starting microservices worker
strange, whats the output of ps aux
inside the container?
Its chowning away 🗡️ but the thing is a) its all owned by root and b) it ran for 3 hrs yesterday and completed... c) v1.107.1-ig297 don't do that. As I mentioned above, this has been a recurring issue for me that it usually took solid couple of minutes to start up immich. After @martabal suggestion to use the same user who mounts nfs share to the filesystem (in my case root) I changed the user, let it chown and now it starts up instantly on v1.107.1-ig297 instead of taking the couple of minutes.
root@c77fac916192:/# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 436 0 ? Ss 10:23 0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service
root 15 0.0 0.0 2800 1664 ? Ss 10:23 0:00 /bin/sh -e /run/s6/basedir/scripts/rc.init top
root 16 0.0 0.0 216 0 ? S 10:23 0:00 s6-supervise s6-linux-init-shutdownd
root 19 0.0 0.0 204 0 ? Ss 10:23 0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3000 -C -B
root 38 0.0 0.0 216 0 ? S 10:23 0:00 s6-supervise svc-microservices
root 39 0.0 0.0 216 0 ? S 10:23 0:00 s6-supervise s6rc-fdholder
root 40 0.0 0.0 216 0 ? S 10:23 0:00 s6-supervise svc-server
root 41 0.0 0.0 216 0 ? S 10:23 0:00 s6-supervise svc-cron
root 42 0.0 0.0 216 0 ? S 10:23 0:00 s6-supervise svc-machine-learning
root 43 0.0 0.0 216 0 ? S 10:23 0:00 s6-supervise s6rc-oneshot-runner
root 47 0.0 0.0 220 0 ? S 10:23 0:00 s6-rc -v2 -u -t 0 -- change top
root 51 0.0 0.0 208 0 ? Ss 10:23 0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s
root 55 0.0 0.0 200 0 ? S 10:23 0:00 /package/admin/s6-2.12.0.2/command/s6-sudoc -e -t 30000 -T 0 -- up 11
root 59 0.0 0.0 212 0 ? S 10:23 0:00 /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --
root 62 0.0 0.0 7340 3712 ? S 10:23 0:00 bash /etc/s6-overlay/s6-rc.d/init-config-immich/run
root 81 0.0 0.0 6936 2816 ? S 10:23 0:00 find /photos -maxdepth 0 ( ! -user abc -o ! -group abc ) -exec lsiown -R abc:abc {} ;
root 85 0.0 0.0 7340 3584 ? S 10:23 0:00 /bin/bash /usr/bin/lsiown -R abc:abc /photos
root 92 11.6 0.1 39664 35200 ? S 10:23 0:02 /usr/bin/find /photos ! -xtype l ( ! -group abc -o ! -user abc ) -exec chown abc:abc {} +
root 249 0.0 0.0 2800 1536 pts/0 Ss 10:23 0:00 /bin/sh -c eval $(grep ^$(id -un): /etc/passwd | cut -d : -f 7-)
root 260 0.0 0.0 7604 4224 pts/0 S 10:23 0:00 /bin/bash
root 361 0.0 0.0 6368 2944 ? D 10:23 0:00 chown abc:abc /photos/thumbs/7edf4625-7fd8-4ded-9f73-383119746d91/c2/7a/c27abde6-3958-4c67-a1cd-ffdf211477b0.webp /photos/thumbs/7edf4625-7fd8-4ded
root 362 0.0 0.0 11104 4864 pts/0 R+ 10:23 0:00 ps aux
root@b56195cb29df:/# ls -lah /photos/thumbs/
total 16M
drwxrwxrwx 10 root root 4.0K May 20 16:16 .
drwxr-xr-x 7 root root 4.0K Jul 11 10:25 ..
drwxrwxrwx 2 root root 288K Oct 1 2023 1ac17131-35fb-45ca-8450-946c68680787
drwxrwxrwx 258 root root 788K Dec 29 2023 21b98a8e-4c28-4ce2-87c6-9cd6f9786aa5
drwxr-xr-x 258 root root 4.0K May 5 15:49 47639fef-eba1-4329-b2ec-543544a1cb16
drwxrwxrwx 257 root root 2.9M Jun 11 11:16 7edf4625-7fd8-4ded-9f73-383119746d91
drwxr-xr-x 47 root root 4.0K May 20 19:51 8b2baceb-13b8-4a2f-b87a-463a3c86eefe
drwxrwxrwx 258 root root 10M Oct 28 2023 bb6df988-8f2a-4ad5-99b5-a1a4f09d1490
drwxrwxrwx 258 root root 1.4M Jan 20 15:54 bf35482e-8ee4-4b6f-a6b8-78faf47c8142
drwxr-xr-x 258 root root 4.0K Dec 30 2023 c6d9dde7-2760-41c1-8833-42eddfbd1055
hmm, I have a similar setup, with an nfs share mounted via systemd; playing around with chown options found some interesting results;
abc@60daa6e0a425:/$ time find /photos -type d -exec lsiown -R abc:abc {} \;
real 0m38.141s
user 0m20.806s
sys 0m6.461s
abc@60daa6e0a425:/$ time find /photos -type d -exec lsiown -R abc:abc {} +
real 0m9.964s
user 0m0.257s
sys 0m2.539s
abc@60daa6e0a425:/$ time lsiown -R abc:abc /photos
real 0m4.958s
user 0m0.050s
sys 0m1.095s
abc@60daa6e0a425:/$
can you edit /etc/s6-overlay/s6-rc.d/init-config-immich/run
(apt update && apt install nano -y
) and change the permissions section to:
# permissions
find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
lsiown -R abc:abc \
"${IMMICH_MEDIA_LOCATION}" \
/config
(on :latest
)
and restart the container and report back
@hydazz Changed run file:
#!/usr/bin/with-contenv bash
# shellcheck shell=bash
# make folders
mkdir -p \
/config/machine-learning/models \
"$IMMICH_MEDIA_LOCATION"
# permissions
find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
lsiown -R abc:abc \
"${IMMICH_MEDIA_LOCATION}" \
/config
Logs:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service init-migrations: starting
s6-rc: info: service init-envfile: starting
s6-rc: info: service init-config-immich: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service init-envfile successfully started
[migrations] started
[migrations] no migrations found
s6-rc: info: service init-migrations successfully started
s6-rc: info: service init-adduser: starting
s6-rc: info: service legacy-cont-init successfully started
╔═══════════════════════════════╗
__ _____ _____ __
/ / |_ _/ ____| / /
/ / | || | __ / /
/ / | || | |_ | / /
/ / _| || |__| | / /
/_/ |_____\_____| /_/
Baseimage from linuxserver.io
╠═══════════════════════════════╣
To support this applications developer(s) visit:
Immich: https://immich.app/docs/overview/support-the-project
╠═══════════════════════════════╣
User/Group ID:
User UID: 0
User GID: 0
╚═══════════════════════════════╝
s6-rc: info: service init-adduser successfully started
s6-rc: info: service init-os-end: starting
s6-rc: info: service init-os-end successfully started
s6-rc: info: service init-crontab-config: starting
s6-rc: info: service init-config: starting
s6-rc: info: service init-config successfully started
s6-rc: info: service init-check-variables: starting
s6-rc: info: service init-crontab-config successfully started
s6-rc: info: service init-check-variables successfully started
s6-rc: info: service init-config-immich successfully started
s6-rc: info: service init-video-immich: starting
s6-rc: info: service init-test-run: starting
s6-rc: info: service init-test-run successfully started
s6-rc: info: service init-config-end: starting
s6-rc: info: service init-config-end successfully started
s6-rc: info: service init-mods: starting
s6-rc: info: service init-video-immich successfully started
s6-rc: info: service init-mods successfully started
s6-rc: info: service init-mods-package-install: starting
s6-rc: info: service init-mods-package-install successfully started
s6-rc: info: service init-mods-end: starting
s6-rc: info: service init-mods-end successfully started
s6-rc: info: service init-custom-files: starting
[custom-init] No custom files found, skipping...
s6-rc: info: service init-custom-files successfully started
s6-rc: info: service init-services: starting
s6-rc: info: service init-services successfully started
s6-rc: info: service svc-server: starting
s6-rc: info: service svc-cron: starting
s6-rc: info: service svc-cron successfully started
Starting api worker
EDIT SUCCESS It started after less than a minute 👯
@hydazz will this change be included in future images? Or is it part of core immich release? I assume I can mount this file to the container for now from the docker host as a half measure? EDIT yup mounting this file from the docker host in the edited version fixes
we have seen issues with chowning the whole photos directory (taking forever to chown it), so we'll see. Out of curiosity, can you change that file to below?:
# permissions
find /photos -maxdepth 0 \( ! -user abc -o ! -group abc \) -exec lsiown -R abc:abc {} +
find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
lsiown -R abc:abc \
/config
and see if the container starts?
Yep it starts instantly now
@martabal would love this change included in future images pleaaaaase 😄 @hydazz you made my day mate, huge thanks to you and @martabal for your help! 🥇
Can you try with this ?
#!/usr/bin/with-contenv bash
# shellcheck shell=bash
# make folders
mkdir -p \
/config/machine-learning \
"$IMMICH_MEDIA_LOCATION"
# permissions
if find "$IMMICH_MEDIA_LOCATION" -maxdepth 0 \( ! -user abc -o ! -group abc \) | grep -q "$IMMICH_MEDIA_LOCATION"; then
echo "**** changing permissions for $IMMICH_MEDIA_LOCATION ****"
lsiown -R abc:abc "$IMMICH_MEDIA_LOCATION"
fi
if find /config -maxdepth 0 \( ! -user abc -o ! -group abc \) | grep -q /config; then
echo "**** changing permissions for /config ****"
lsiown -R abc:abc /config
fi
find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
I'll open a PR tonight if that works for you.
@martabal bet you to the punch 😁
@martabal bet you to the punch 😁
Ahah, nice
Can you try with this ?
#!/usr/bin/with-contenv bash # shellcheck shell=bash # make folders mkdir -p \ /config/machine-learning \ "$IMMICH_MEDIA_LOCATION" # permissions if find "$IMMICH_MEDIA_LOCATION" -maxdepth 0 \( ! -user abc -o ! -group abc \) | grep -q "$IMMICH_MEDIA_LOCATION"; then echo "**** changing permissions for $IMMICH_MEDIA_LOCATION ****" lsiown -R abc:abc "$IMMICH_MEDIA_LOCATION" fi if find /config -maxdepth 0 \( ! -user abc -o ! -group abc \) | grep -q /config; then echo "**** changing permissions for /config ****" lsiown -R abc:abc /config fi find /app/immich -path "*/node_modules" -prune -o -exec chown abc:abc {} +
I'll open a PR tonight if that works for you.
@martabal works, the same as @hydazz solution
On latest version v1.107.2-ig299, Immich doesn't start, just stays like the in the screenshot