search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
39.44k stars 1.86k forks source link

Storage template migration - unexpected behaviour #8187

Open lmgithu opened 3 months ago

lmgithu commented 3 months ago

The bug

When I set a new template for the folder structure (switching from yyyy-mm-dd to yyyy-mm only) and run the storage template migration job, I see that 1) the job finished running 2) 80% of the folders now have a mix of yyyy-mm-dd and yyyy-mm folders with data in them).

I wanted to check why immich is not capable of transforming all folders and why there are leftovers? My other contern is whether this is breaking viewing the images and/or the database.

Desired outcome:

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.98.2

Version of Immich Mobile App

v1.98.2

Platform with the issue

Your docker-compose.yml content

standard

Your .env content

standard

Reproduction steps

1. set a new storage template
2. run the template migration job
3. not all folders are getting migrated, it is absolutely random

Additional information

No response

lmgithu commented 3 months ago

some additional observations:

If i start another migration to migrate thumbnails for assets and faces to the latest folder structure, that job always runs on ALL my assets (100K+)... each time i see the huge number of assets that go down to zero but it seems it is not getting processed in reality as each time when i start the job it shows same high number.

alextran1502 commented 3 months ago

Can you help double-check the files in the older format yyyy-mm-dd to see if the path corresponds to the file's path that is shown in the UI?

image

lmgithu commented 3 months ago

I think the issue is now even bigger, I tried to check a file in that folder and one of them is there on the file system but NOT on the UI.. I don't see these images anymore :( In another case I do see as you do, it is on the UI as well and it still refers to the old folder (attached screenshot below)

Screenshot 2024-03-22 at 16 03 00

lmgithu commented 3 months ago

If you need, I can share the logs and/or the file itself that it did not process. Let me know how I could help.

alextran1502 commented 3 months ago

NOT on the UI.

Can you use the search for file name to find that file to see if it returns in the search result. Please attach the screenshot of the file on the file system as well as the search result if you can

alextran1502 commented 3 months ago

Also logs, on the server when you run the migration would be helpful, you can increase the log level to debug in the Server settings in the Administration page

lmgithu commented 3 months ago

Will do and report back in an hour. How can I download/extract the log file?

alextran1502 commented 3 months ago

From the terminal, run docker logs immich_server or docker logs immich_microservices

lmgithu commented 3 months ago

hi Alex,

Here are the results: searching for the image name on UI: no result. I can't take a screenshot as there is no results, it just shows my photo feed (I confirm it works for other image names). Screenshot of file system/location of the image: attached.

Screenshot 2024-03-22 at 19 30 00

Log:

immich_server_1            | [Nest] 6  - 03/22/2024, 6:25:05 PM   DEBUG [JobService] Handling command: queue=storageTemplateMigration,force=false
immich_redis_1             | 15:C 22 Mar 2024 18:05:08.034 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:05:08.107 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:10:09.036 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:10:09.037 * Background saving started by pid 16
immich_redis_1             | 16:C 22 Mar 2024 18:10:09.161 * DB saved on disk
immich_redis_1             | 16:C 22 Mar 2024 18:10:09.162 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:10:09.239 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:15:10.025 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:15:10.025 * Background saving started by pid 17
immich_redis_1             | 17:C 22 Mar 2024 18:15:10.052 * DB saved on disk
immich_redis_1             | 17:C 22 Mar 2024 18:15:10.052 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:15:10.126 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:20:11.071 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:20:11.072 * Background saving started by pid 18
immich_redis_1             | 18:C 22 Mar 2024 18:20:11.098 * DB saved on disk
immich_redis_1             | 18:C 22 Mar 2024 18:20:11.098 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:20:11.172 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:25:12.096 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:25:12.096 * Background saving started by pid 19
immich_redis_1             | 19:C 22 Mar 2024 18:25:12.120 * DB saved on disk
immich_redis_1             | 19:C 22 Mar 2024 18:25:12.120 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:25:12.196 * Background saving terminated with success
immich_machine-learning_1  | [03/22/24 18:25:17] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:27] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:37] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:47] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:57] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:07] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:17] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:27] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:37] DEBUG    Checking for inactivity...
alextran1502 commented 3 months ago

Just to make sure, can you drag-and-drop that file on the web to see if it successfully upload or it will say duplicated

lmgithu commented 3 months ago

I tried, it said "skipped 1 duplicated asset"

alextran1502 commented 3 months ago

So it should be on the server, could it be in the archive view somehow?

lmgithu commented 3 months ago

No, it is not there in the archive :(

alextran1502 commented 3 months ago

Can you confirm the visual of the image on that date,not just the file name?

lmgithu commented 3 months ago

Yes I can, I can't see this image visually on the UI, despite it is on the file system.

alextran1502 commented 3 months ago

hmm strange, can you show me the search query you put in to search for the file name?

lmgithu commented 3 months ago

here you are:

Screenshot 2024-03-22 at 20 13 45

lmgithu commented 3 months ago

this works for any other filename btw and I can see the results

alextran1502 commented 3 months ago

Try 5841? and did you tick on the archive button at the very bottom of the page?

lmgithu commented 3 months ago

Do you know why certain folders/files were ignored? this is how my folders look like:

Screenshot 2024-03-22 at 20 16 07

Interestingly, until 2010 every folder looks OK... after that, it is fully random and mixed.

alextran1502 commented 3 months ago

I am not sure, we will have to look at this more in-depth. Thanks for the report

lmgithu commented 3 months ago

Could I somehow force process folders again?

lmgithu commented 3 months ago

or at least "delete" these files somehow so i could reupload them?

alextran1502 commented 3 months ago

Can you check the Trash page to see if it is in it?

lmgithu commented 3 months ago

WIN, it was in the Trash! Sorry, this was such a noob issue. Now we "just" have the folder structure issue that it is fully random, I think there are no missing images. Do you have any guess what might be the issue and how to fix it?

alextran1502 commented 3 months ago

Do you have any guess what might be the issue and how to fix it?

The issue is that we don't move files that are already in the trash :P So it is working as expect

lmgithu commented 3 months ago

no, it was a SINGLE image... not all of them.

lmgithu commented 3 months ago

99% of the files that were not moved are NOT in the Trash

alextran1502 commented 3 months ago

Then we will need to circle back and look at this!

lmgithu commented 3 months ago

Thank you! Is there any way to "force" the migration again?

lmgithu commented 3 months ago

e.g. if i switch back to yyyy-mm-dd again and switch again to yyyy-mm afterwards? or via CLI?

alextran1502 commented 3 months ago

Besides clicking on the template migration job, nothing else can be done. Grab the server logs when you click on that button would be helpful as well

On Fri, Mar 22, 2024 at 2:37 PM lmgithu @.***> wrote:

Thank you! Is there any way to "force" the migration again?

— Reply to this email directly, view it on GitHub https://github.com/immich-app/immich/issues/8187#issuecomment-2015784865, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGONL7XHXBUFQKRR6RHAM3TYZSB75AVCNFSM6AAAAABFDCMSJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJVG44DIOBWGU . You are receiving this because you commented.Message ID: @.***>

-- Alex Tran

danieldietzler commented 3 months ago

Just to make sure, are there assets in those leftover folders or are they all empty? In any case re-running the storage migration will attempt to fix potential issues, so that is certainly worth a shot!

lmgithu commented 3 months ago

hi Daniel,

Unfortunately the folders are not empty, there are images in them. I run the migration job again for the original template format (yyyy-mm-dd) and it seems it reversed it to that properly, all yyyy-mm folders disappeared. So for that it seems to be working fine, but I don’t dare trying again for the yyyy-mm format again not to screw something. It would be great if somebody could check what might be the root cause so people could migrate properly.

If I can provide any further information, please let me know I will try to help.

danieldietzler commented 3 months ago

I am somewhat certain that running the migration on yyyy-mm again would have solved the issue as well, but of course I can't tell for sure. Many filesystem moves just happen to be flaky sometimes (most of the time actually lol).

zackpollard commented 3 months ago

Hey! Would be great to get some more information on what's actually happening here. If you were willing to change the storage template structure again then it would be great if you could follow these steps and then provide us with the logs for your microservices instance.

1.) Change storage template to yyyy-mm format again, let the jobs finish. 2.) Check that the assets didn't all move correctly 3.) Try running the storage template migration job again manually 4.) Provide the full logs for the microservices container after this second job completes

Also if you are able to provide an example image that fails to move, that would be great. Could you also share what filesystem you're using to store the asset files?

Cheers!

danieldietzler commented 3 months ago

@lmgithu ?