Open Dunky-Z opened 1 month ago
mertalev
commented
1 month ago How much RAM does the server have, and how much is used just after starting thumbnail generation? An increase in memory usage for some period of time during thumbnail generation is normal because of memory fragmentation, but it should plateau after a certain point. If you have any RAW images, they will amplify this effect since they require much more memory.
Dunky-Z
commented
1 month ago Thank you for your response. The server has a total of 16GB of RAM, and its typical daily memory usage hovers around 6GB. After initiating thumbnail generation, it doesn't immediately consume all memory but gradually fills it up until the program crashes. Your mention of RAW images did remind me that the majority of my gallery consists of RAW files. Is there a way to prevent excessive memory usage in this scenario? I have tried limiting the microservice using Cgroups, but when the limit is exceeded, the microservice restarts. Since I bind the Cgroup settings to the PID, once the microservice restarts, PID changed, the previous Cgroup configurations become ineffective.
Dunky-Z
commented
1 month ago Given the server's average performance, I have configured all my JOBS to use a single thread, and only one JOB is executed at a time. In theory, processing a single RAW file shouldn't require such an excessive amount of memory.
mertalev
commented
1 month ago Hmm, that much of an increase is unexpected. It could be related to the issue behind #6542, or possibly #4391. Are there any errors in the microservices logs before the OOM error?
As far as limiting memory usage in the meantime, you can set a limit through Docker, which will force the container to be restarted after reaching a certain usage.
Dunky-Z
commented
1 month ago Below are some logs from the microservices; perhaps they might provide some insight:
[Nest] 7 - 05/13/2024, 11:34:38 AM LOG [EventRepository] WebSocket server initialized.
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: The input file contains an unsupported image format
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Error: The input file contains an unsupported image format
at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
at MediaRepository.resize (/usr/src/app/dist/repositories/media.repository.js:76:14)
at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:156:48)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:134:29)
at async /usr/src/app/dist/services/job.service.js:149:36
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Affected Job ID:
{
"id": "5802ddbb-4e01-47fb-b21b-73bc0361f9e8"
}
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
Built with gcc 12 (Debian 12.2.0-14)
Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
Built with gcc 12 (Debian 12.2.0-14)
Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input
at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
at ChildProcess.emit (node:events:518:28)
at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Affected Job ID:
{
"id": "ee18e5e8-5430-4fcb-aa10-ef4170e2b30e"
}Limiting the container's resource usage has indeed helped me; it prevents my server from crashing. However, upon examining the microservices' logs, I noticed frequent restarts, indicating that the microservices are indeed exceeding the memory limit. Even after setting a 4GB memory limit, it's still insufficient.
Here's the relevant part of my Docker Compose configuration:
immich-microservices:
container_name: immich_microservices
image: altran1502/immich-server:${IMMICH_VERSION:-release}
command: ['start.sh', 'microservices']
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- "/root/sharedfolder/syncthing/Photo_Album:/mnt/media/Photo_Album:ro"
- /etc/localtime:/etc/localtime:ro
devices:
- /dev/dri:/dev/dri
env_file:
- .env
depends_on:
- redis
- database
restart: always
deploy:
resources:
limits:
cpus: '2'
memory: 4G[Nest] 7 - 05/13/2024, 11:35:19 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:33 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:45 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:58 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:36:11 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:36:53 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:06 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:19 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:34 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:47 AM LOG [EventRepository] Initialized websocket serverI'm still hoping to identify and resolve the root cause entirely. If more information is needed, please let me know.
On a side note, how can I persistently save Immich logs? The log path isn't mentioned in the user documentation, so I haven't mapped any volumes, resulting in log loss upon container restarts.
Thank you very much！
ceebu
commented
1 month ago I have the same problem (i5 8th gen server, running OMV 6) - am also Looking for a easy way to save immich logs -
RazerProof
commented
1 month ago
I can also confirm this issue is affecting the installation on my NAS. Limiting memory causes frequent restarts of the microservices container. I also tried setting the Generate Thumbnails Concurrency to 1, but it didn't help. I found the NAS started overutalising the drives as they were at 100% utilisation. I imagine it was paging out memory to disk; CPU utilization was around 40%. I have increased the RAM in the NAS to 32GB, which has solved the problem for me, and the microservices container now runs constantly, with no restarting at all. The microservices container stabilised at around 28GB, the NAS memory resource monitor shows almost all of that usage is cache memory, the container limit is set to 4GB.
After restarting the container, it takes about 15-20min to stabilise at this level. Disk usage is now around 40% and CPU is around 80%, generating thumbnails at around 30/min. This is with no other jobs running.
I have around 230k assets in the external library, 50%jpg, 50%RAW. I also tried this in docker on a server (TrueNAS scale) and experienced the same issue and outcome.
Overall, I love the way Immich is going. It is going to be a great product. Well done to everyone involved, and thanks for the great work.
mertalev
commented
1 month ago Thanks for the detailed info!
/tmp configured to be in-memory?RazerProof
commented
1 month ago How many of those assets are RAW, if any? 50% of the images are RAW around 50GB each.
Do you have /tmp configured to be in-memory? No /tmp configured.
Is the increase completely linear, or are there spikes? Mostly linear overlayed with a small saw tooth shape.
I just restarted the container. That smooth section is unusual, and it will go back to the sawtooth. Any errors in the logs? No Errors, running smoothly.
RazerProof
commented
1 month ago Below is an example of the errors I was receiving on the NAS when 4GB was installed and getting constant restarts of the microservices container.
mertalev
commented
1 month ago I made a test image for microservices with a possible fix: ghcr.io/immich-app/immich-server:pr-9665. Would you be able to change your image to that and see if it affects RAM usage?
RazerProof
commented
1 month ago Not good news I am afraid. The immich_microservices fails.
immich_microservices
date stream content
2024/05/23 01:09:52 stderr Microservices worker exited with code 1
2024/05/23 01:09:52 stderr }
2024/05/23 01:09:52 stderr routine: 'parserOpenTable'
2024/05/23 01:09:52 stderr line: '1381', 2024/05/23 01:09:52 stderr file: 'parse_relation.c', 2024/05/23 01:09:52 stderr constraint: undefined, 2024/05/23 01:09:52 stderr dataType: undefined, 2024/05/23 01:09:52 stderr column: undefined, 2024/05/23 01:09:52 stderr table: undefined, 2024/05/23 01:09:52 stderr schema: undefined, 2024/05/23 01:09:52 stderr where: undefined, 2024/05/23 01:09:52 stderr internalQuery: undefined, 2024/05/23 01:09:52 stderr internalPosition: undefined, 2024/05/23 01:09:52 stderr position: '128', 2024/05/23 01:09:52 stderr hint: undefined, 2024/05/23 01:09:52 stderr detail: undefined, 2024/05/23 01:09:52 stderr code: '42P01', 2024/05/23 01:09:52 stderr severity: 'ERROR', 2024/05/23 01:09:52 stderr length: 113, 2024/05/23 01:09:52 stderr }, 2024/05/23 01:09:52 stderr routine: 'parserOpenTable'
2024/05/23 01:09:52 stderr line: '1381', 2024/05/23 01:09:52 stderr file: 'parse_relation.c', 2024/05/23 01:09:52 stderr constraint: undefined, 2024/05/23 01:09:52 stderr dataType: undefined, 2024/05/23 01:09:52 stderr column: undefined, 2024/05/23 01:09:52 stderr table: undefined, 2024/05/23 01:09:52 stderr schema: undefined, 2024/05/23 01:09:52 stderr where: undefined, 2024/05/23 01:09:52 stderr internalQuery: undefined, 2024/05/23 01:09:52 stderr internalPosition: undefined, 2024/05/23 01:09:52 stderr position: '128', 2024/05/23 01:09:52 stderr hint: undefined, 2024/05/23 01:09:52 stderr detail: undefined, 2024/05/23 01:09:52 stderr code: '42P01', 2024/05/23 01:09:52 stderr severity: 'ERROR', 2024/05/23 01:09:52 stderr length: 113,
mertalev
commented
1 month ago Hmm, that error is about connecting to Postgres, not related to thumbnail generation.
RazerProof
commented
1 month ago Yeah... Something may have gone wrong with the deployment. Let me do some more testing, and maybe get some sleep :-).
mertalev
commented
1 month ago So sorry! I based that branch off of the latest release, but it turns out that main gets merged into it anyway when the image is built. The error is probably because of that.
You can either wait for the next release to get things back up or restore from a backup. (It's also possible to mess with it more to get it back up, but I think these options are safer.)
RazerProof
commented
1 month ago Thanks for letting me know and I can confirm I am getting the same result.
It is working, and there are no errors in the log; it's just hungry :-)
I will wait for the next release and re-test
Thanks for all your great work.
ThisIsCyreX
commented
3 weeks ago Same issue here, worked great for some time until I noticed the slow ssh and docker container restarting on its own. I re-created all thumbnails and thought maybe that will help, The job ran without issue always ~1gig memory. From 200k, the last 60 or so thumbnails are the problem I guess. Fills the memory and the log says this:
[Nest] 18 - 06/12/2024, 9:53:14 PM ERROR [Api:ExceptionsHandler~wx0506ns] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
at Object.onceWrapper (node:events:633:28)
at Connection.emit (node:events:519:28)
at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
at Socket.emit (node:events:519:28)
at TCP.<anonymous> (node:net:338:12)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
Detected CPU Cores: 6
Starting api worker
Starting microservices worker
[Nest] 7 - 06/12/2024, 9:53:44 PM LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7 - 06/12/2024, 9:53:45 PM LOG [Microservices:SystemConfigService] LogLevel=verbose (set via system config)The job will stay for ca 10 minutes on active 1; waiting: 0, fill the memory without posting anything, except my client viewing the page.
/edit: I removed a read-only external library and now the job is fine again. Need to check what asset(s) causes the leak. /edit2: Found one PSD file that crashed my server. 208 MP, 11792 x 17649, 47,5 MiB - Was 2gig before redacting (but still crashing). I can upload it to immich, but the thumbnail job won't finish. File: immich-stuck-thumbnail_PSD.zip
The bug
Today, I encountered the same issue #5283 . After bulk importing around 10,000 photos(By setting the external library), I suddenly couldn't remotely connect to my NAS host. Initially, I suspected that the ML service was causing high memory usage during facial recognition, so I forcibly restarted the NAS and reconfigured concurrency settings, setting all jobs to run on a single thread. I didn't start all the jobs at once; instead, I sequentially ran FACE DETECTION and then GENERATE THUMBNAILS. There were no abnormalities during the FACE DETECTION process, indicating that the issue wasn't caused by the ML operations. After completing all FACE DETECTIONS, I initiated the GENERATE THUMBNAILS task, which ran slowly due to the large image sizes. Without continuous monitoring, upon returning to check on the NAS, I found it was unreachable remotely, with the system log displaying the following error messages:
This suggests that there might be a memory leak issue within the immich_microser service. I would greatly appreciate any advice on how to address this problem.
The OS that Immich Server is running on
Debian(OMV)
Version of Immich Server
OpenMediaVault 5
Version of Immich Mobile App
v1.103.1
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Relevant log output
No response
Additional information
No response