Open aks-cadesign opened 1 month ago
Any thoughts on this @zackpollard?
How big is your external library? Count of assets is the main thing
19.903 files, 1.450 folders, 145GB
takes around 6 minutes until it crashes:
not a lot of files and folders for this error. i can only share my experiences. at the moment i have 10 libraries (~600 folders each) and 1.35 million assets. on a system with 8gb ram i had problem if a single library gets around 250k+ assets. so be sure you are at ~200k and limit concurrent worker threads for library to 1! i then had never the "JS heap out of memory" again. i have no special variables @.env etc.. only default stuff.
because your numbers are much smaller than that, i dont have a clue what the problem is. but try concurrent librarys workers set to 1. i dont know why default setting is 5. in my opinion its too high, because a lot of memory stuff is not fixed at the moment. and even for my over millions assets a setting of 1 is enough.
I moved to a different machine with more RAM and changed NODE_OPTIONS environment variable to: --max-old-space-size=49152 and got a new error this time
node:internal/event_target:1094
process.nextTick(() => { throw err; });
^
RangeError [Error]: Map maximum size exceeded
at Map.set (
now running version v1.111.0 RAM usage peaked around 42GB.
concurrent library workers has been set to 1 even before opening this issue.
another thing I noticed: nothing is being added to generate thumbnails or other jobs and it's just the 1 library job running.
Could you please run the following command in the folder your library is in on the host machine and post the result here or DM it to me on discord?
ls -lR /path/to/folder | grep '^l'
I'm facing the same issue with an immich-server
limited to 3GB and with the external library with around 150k assets :/
@kdybicz Are you on 1.113.0 or later? That release lowered RAM usage when scanning large libraries.
@mertalev I've tested it yesterday with 1.113.1
The bug
library job failed after adding external library
The OS that Immich Server is running on
Ubuntu 24.04 LTS
Version of Immich Server
v1.108.0
Version of Immich Mobile App
-
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Relevant log output
Additional information
at first, it crashed around 4.3GB RAM usage, so I added 'NODE_OPTIONS=--max-old-space-size=8196' env variable but I get the same outcome with increased memory usage (9.4GB).