Closed CorentinGrard closed 11 months ago
alextran1502
commented
1 year ago I've seen a lot of reports lately of this issue when running on the TrueNas scale environment, I am not sure what is the root issue.
If you can add this to the VM's environment variables that you are running Immich
NODE_OPTIONS="--max-old-space-size=8192"
CorentinGrard
commented
1 year ago I've tried that but the container still crash after eating lot more memory
<--- Last few GCs --->
[7:0x3efd6310000] 950404 ms: Scavenge 8056.3 (8222.7) -> 8042.9 (8222.7) MB, 26.43 / 0.00 ms (average mu = 0.059, current mu = 0.010) allocation failure;
[7:0x3efd6310000] 950528 ms: Scavenge 8057.9 (8222.7) -> 8046.2 (8223.5) MB, 39.88 / 0.00 ms (average mu = 0.059, current mu = 0.010) allocation failure;
[7:0x3efd6310000] 1073056 ms: Mark-Compact 8061.9 (8225.2) -> 8044.0 (8225.7) MB, 120888.41 / 0.00 ms (average mu = 0.038, current mu = 0.017) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xc99960 node::Abort() [immich_microservices]
2: 0xb6ffcb [immich_microservices]
3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
5: 0x10d06a5 [immich_microservices]
6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7ff8aca99ef6
jrasm91
commented
1 year ago Does anything change if you turn off reverse geocoding?
CorentinGrard
commented
1 year ago Disabling reverse geocoding solved the issue and now the thumbnails generation can go through ! Any idea of what is happening and how I could turn it back on without crashing ?
jrasm91
commented
1 year ago Suuuuucks, no. Probably something dumb though. We use a 3rd party implementation and it seems to be the bane of my existence.
CorentinGrard
commented
1 year ago Alright, at least it's now my instance is usable. Thank you for the help !
yodatak
commented
1 year ago Got the same issue in k3s with helm chart
flymin
commented
11 months ago Does anything change if you turn off reverse geocoding?
Same problem here on truenas. Turning reverse geocoding off helps. Why cannot enable this feature?
jrasm91
commented
11 months ago We use a third party library for this and it has lots of issues. We're actually rewriting it from scratch into immich right now because of that. Once merged it should be possible to use the feature reliably.
zackpollard
commented
11 months ago In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!
The bug
Hi,
I installed a week ago immich and ingested the data with immich-go. So far no problems, but then come the time to run the jobs. At first I through it was quite slow but after some investigation I find the container immich-microservices crashing with an out of memory every 2 minutes. I have tried to give it more memory as this post suggest but it still crash as early as without this flag and just take more time to fillup the ram and crash. I also took down in the settings the machine learning part as there was some error msg in the log related to this but it didn't have any impact on the problem.
Immich is running under a Nixos VM and the host is a Truenas scale and the immich upload folder is a NTF drive.
Log of the immich-microservice container with log set to debug
The OS that Immich Server is running on
NixOS 23.05
Version of Immich Server
1.85.0
Version of Immich Mobile App
1.85.0
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Additional information
No response