mertalev
commented
6 months ago Does the crash occur when transcoding? If so, this sounds like a bug with FFmpeg you may want to report (including the corrupted file to reproduce the issue).
Perhaps a good practice would be to track the corrupted files or those that are otherwise unsuccessful at being processed. After a certain number of attempts they are marked as "problem files" and ignored. Otherwise it seems that filtering a huge number of files down to a small kernel of problem files, and then retrying thumbnail/metadata/face recognition, etc on those problem files over and over and over again, seems a surefire way to surface problems and bugs with corrupted files, and then have those problems and bugs recur every single day as the library is reprocessed.
We actually don't retry failed jobs. What I think is happening is that as the job waits for transcoding to complete, the whole container crashes from OOM without the job actually being marked as failed.
bhugh
The bug
TL;DR of below is, microservices consumes all available memory (no matter how much is provided - up to at least 12 or 15GB), locks up the machine, then crashes the container when it encounters a particular corrupted file,
IM000114.AVI- and perhaps other similar corrupted files.I realize that simply getting rid of that corrupted file is one solution to this issue, but it still seems that Immich shouldn't use all available memory, lock up the entire machine, crash the container, etc etc, even when encountering a corrupted file. Most corrupted files are handled just fine.
Also: Processing ~250,000 files successfully, there are now a couple dozen corrupted files remaining. Since they are corrupted, they never process metadata or thumbnails successfully. Every time these jobs are re-run, it attempts the corrupted files again.
Since one (or perhaps more) of the corrupted files are causing microservices to crash, these crashes are happening regularly now.
Perhaps a good practice would be to track the corrupted files or those that are otherwise unsuccessful at being processed. After a certain number of attempts they are marked as "problem files" and ignored. Otherwise it seems that filtering a huge number of files down to a small kernel of problem files, and then retrying thumbnail/metadata/face recognition, etc on those problem files over and over and over again, seems a surefire way to surface problems and bugs with corrupted files, and then have those problems and bugs recur every single day as the library is reprocessed.
LONG VERSION:
For a while now I've noticed microservices will - once in a while! - fill up all available memory and swap space. This will lock up the machine and the only solution is to reboot.
I tried various options to limit memory for microservices - limited to 2-3-4 GB it ran great but couldn't complete quite a lot of tasks. (The machine has 6GB memory + 12GB swap available.) Increasing to say 8 or 12GB allowed most all jobs to complete but every couple of days it would still hit this "fill memory" bug and the machine would be locked up.
I can watch the problem develop using a tool like htop. Most of the time everything moves along with not problems, maybe 2-4GB memory used and no swap. But when this problem begins, the memory will gradually start to fill up, then swap memory. The only large user of the memory is microservices. docker stats just locks up while this is happening and doesn't give any useful data (it literally goes blank). If microservices memory is not limited it will continue until both real and swap memory and completely filled. At this point the machine is locked up and won't respond at all. The only way out is to reboot.
If microservices memory is limited, then it appears to crash at some point when that memory is exhausted. The rest of the machine returns to normal and microservices restarts after a couple of minutes and returns to normal.
The settings in the attached docker compose file give the memory settings that have worked for me - they seem to allow all necessary jobs to complete, but when this runaway memory issue occurs, the max memory consumption allowed is small enough to keep the machine from totally locking up.
Limiting memory using Docker Compose did not seem sufficient to completely solve the problem. Adding the line to limit max-old-space-size to 5096 solved the problem - in the sense that it still happens and still fails within microservices (its memory entirely fills up), but that runaway memory condition within the microservices container doesn't bring down the whole machine. The microservices container seems to crash and recover.
I put the same memory and max-old-space-size limits on both microservices and machine learning. However, it appears to mainly by microservices that causes the runaway memory problem.
The problem seems to be (predictably/repeatably) caused by certain specific corrupted files. I've processed around 250,000 files, and most now have metadata & thumbnails generated. Of course there are a few that are corrupted for one reason or another. Those make up a large proportion of the remaining files without thumbnails, so now it is possible to cause the problem to recur simply by "generating thumbnails - missing".
Most of the corrupted files seem to process normally and simply fail. But certain of the files cause this runaway memory problem to happen.
The OS that Immich Server is running on
Ubuntu 22.04 (WSL under Windows 10)
Version of Immich Server
v1.94.1
Version of Immich Mobile App
(not relevant)
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Additional information
This is the microservices container log for right before and then right after one of these memory events.
Starting at 6:40:43 you can see it is processing a large number of files with "corrupt header" and other such problems. But no major problems.
Starting at 6:40:45 it encounters
Photowise-Images/2002-12 (Dec)-15/IM000114.AVI: Invalid data found when processing input. This seems to be where the problem lies.This is the same time when I see memory rising in
htopanddocker statslocks up. The machine becomes quite unresponsive.After a minute or two the memory is released, the rest of the machine returns to normal responsiveness. No log messages from microservices. The microservices container appears to have crashed and is restarting.
The next log messages from microservices are at 6:44:05 AM and it looks like the container is restarting and reinitializing:
I have tried this a few times now and always when it reaches
IM000114.AVIis the time when it locks up and crashes.