search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
39.06k stars 1.85k forks source link

Files wrongly reported as untracked #7436

Open bo0tzz opened 4 months ago

bo0tzz commented 4 months ago

Discussed in https://github.com/immich-app/immich/discussions/7435

Originally posted by **stumpylog** February 26, 2024 After updating to 1.95.1, approximately half my images are reported as being "untracked" on the repair page. However, I can locate the images in question from the main timeline and they are viewable and downloadable. Taking a single image as an example: The repair page lists `/usr/src/app/upload/library/me/2023/01-January/2023-01-14/PXL_20230114_222814182.jpg` as untracked. The image page lists the file as `upload/library/me/2023/01-January/2023-01-14/PXL_20230114_222814182.jpg` The image downloads as PXL_20230114_222814182.jpg and the SHA matches the stored image on the server.
stumpylog commented 4 months ago

Perhaps related: #7333, however these images are not archived and never have been (I don't use that feature).

There are 6239 files being reported as untracked, though I've only checked a few at this point I assume the issue is the same with all of them.

abangtor commented 4 months ago

Same Issue here. Repair is reporting 36624 untracked files of a total of 40735 stored pictures...

bshep commented 4 months ago

I have 64940 untracked files... the repair page is barely able to load and display the list, i'm not sure how to clean this up... they all seem to be thumbnails....

alextran1502 commented 4 months ago

@bshep Hi, just for some data point, how old is this Immich instance?

stumpylog commented 4 months ago

My instance is roughly 2 years old at this point, those I recently re-created all the thumbnails (probably withing the last 6 months) to clear the issues list of umtracked files, which were all thumbnails then.

This time, some actual assets are reported as untracked

bshep commented 4 months ago

@bshep Hi, just for some data point, how old is this Immich instance?

About 5days, I just started using it as an offline repository of our photos. Most of the photos are in external libraries.

bshep commented 4 months ago

Is there any way to get a list of the untracked files as json or plain file? If so I can grep the output and rm the untracked thumbs and clear out most if not all the untracked files. As it is the 'repair' page can barely be loaded as it is very large.

Allram commented 4 months ago

I had 16k untracked files (which can't be true as they can be found in Immich itself), then i reran all the Thumbs. Now i have 55k assets which are untracked of which ~40k is thumbs.

1.97.0

adminmat commented 4 months ago

I'm showing 52K Untracks Files. Mostly thumbs and encoded videos. Hosting 33K photos. v1.97.0

ejbastine commented 3 months ago

Is there any way to get a list of the untracked files as json or plain file? If so I can grep the output and rm the untracked thumbs and clear out most if not all the untracked files. As it is the 'repair' page can barely be loaded as it is very large.

On the repair page, click "export" in the top right. Or you can export via api at http://your_immich_address/api/audit/file-report

I'm having a related issue where the values on this repair page aren't updating. None of the extra files report, nor the orphan files, actually exist in my filesystem. When I try to run the Remove Offline Files script (https://immich.app/docs/guides/remove-offline-files) it gives me a 400 error on the orphan files, which seems to be correct as I can't find any of the orphans in the immich postgres. I've removed the untracked files but the file report won't update, even after restarting all containers, refreshing, etc.

Chuckame commented 3 months ago

Same

TransRapid commented 1 month ago

Export o text, rplace text that replaces the path to your directory, with "rm" in front of it. Save the file as a simple bash script, and then run that as your immich user, of course verify what the files are before you run it.

malavolti commented 1 month ago

I think to have the same issue.

IMMICH-offline-paths

IMMICH-untracked-files

And I suspect that they are appeared from the usage of the External Libraries in my first instance.

Can you help me to understand how to clean this?

Thank you!

Chuckame commented 1 month ago

Be REALLY attentive when removing files, I've loosen a lot of photos when a bug occurred with thumbnails and made a high amount of untracked files (700k+ for 35k assets).

malavolti commented 1 month ago

Dear All,

I've used the immich-tools to fix the untracked-files, by Python script for offline paths does not work, probably due a wrong immich API usage.

From the Immich Tools I read:

Immich directory paths are maintained at destination directory in order to distinguish between files located "thumbs" diectory (files generated by Immich and are not unique media) and "library or upload" directories (files which may be unique, and may need to be reimported).

This means that all "thumbs" files are created by Immich and "library or upload" files have to be re-imported. I saw that all not "thumbs" files(19) that the script told me to re-import were already into my immich instance.

I didn't lose any file, apparently.

I hope this can be useful for others.

Have a nice day! Marco

pfouto commented 1 month ago

Having a similar issue. Had around 19k untracked files (with 26k pictures), most of them thumbnails but also lots of pictures.

Used immich-tools to move the untracked files away with the intent of re-uploading them, and the result was both all of those files being now shown as offline paths and 11k new files appearing as untracked.

I restored the files, which cleared all "offline paths" and confirmed that the files showing as untracked (at least the few ones I checked) are indeed viewable and downloadable from the timeline.

I also noticed that the number of untracked files is different whenever I open the repair page, varying from ~16k to ~22k.

If it helps, my instance is around 1 year old, and I have never used external libraries.