Closed psla closed 4 days ago
Wondering if periodic library scanner would re-queue files that are already in the queue.... Leading to processing them multiple times.
I just disabled it to see if it will help.
Looks like you opened this issue in the wrong repository.
Pre-submission Checks
Describe the bug
I refreshed my library in immich after fixing issue with glob patterns. Unfortunately, the thumbnails do not seem to get processed, or get processed extremely slowly (after a week I am still 300k short of all files, and I have been 300k short 5 days ago too).
While some progress is being made, Immich attempted to parse a folder containing 11k files approx 130k times. Here is a quick analysis of a single file,
/mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG
There are 32 entries for this file in the docker log:
It looks like this file was queued for metadata extraction 5 times
And the preview was also generated 4 times
The log file from docker has now over 4GB with 13M lines.
It seems like immich is processing some of the same files multiple times and not getting through the backlog of the files to process.
logs.gz
Have you seen this before? Is there anything else I can do to help diagnose the issue?
It does seem like Immich is making some progress, as the number of "outstanding" files reduced in the past 3 days, but it feels like it's making a progress slowly, processing each file multiple times.
Expected behavior
Process each file once
Steps to reproduce the issue
not sure.
1) I changed the "excluded path" patterns in my external library (one that was processed before) 2) And then I refreshed (scan library) operation
I've waited a week and it seems like immich got stuck, not making much progress, and recent files do not have thumbnails generated for them (some do, some don't)
Screenshots
Screenshot today:
Screenshot 3 days ago
Edit! Screenshot now (a few hours later). The backlog is back to 320k items. and FWIW no Library tasks or metadata tasks are being performed (i.e. I am not adding tens of thousands of new files)
Logs
logs.gz
Happy to provide more, the file is massive though (4GB, might compress to a few hundred megs)
Client version number
n/a (web client)
Desktop environment (Linux only)
Linux 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux
Client package version and origin (Linux only)
No response
Installation path (Windows only)
No response
Server information
1.115.0, docker
Additional context
No response