CLI storage purge not working

[EinAeffchen]

Hey, I recently removed some files from my library and noticed they kept showing up on the website anyways. After some research I found the gallery.js storage purge command, if I understand correctly that is supposed to remove orphaned images right?

When I run the command it fails to delete anything due to a bug though: [2024-10-14 08:24:52.089]: storage.purge warn Could not remove orphan file ff/5f/9a8df41d85f158a33f78c6d572be938d6b6c-image-preview-320.jpg: ReferenceError: options is not defined [2024-10-14 08:24:52.090]: storage.purge warn Could not remove orphan file ff/5f/9a8df41d85f158a33f78c6d572be938d6b6c-image-preview-800.jpg: ReferenceError: options is not defined [2024-10-14 08:24:52.091]: storage.purge warn Could not remove orphan file ff/5f/9a8df41d85f158a33f78c6d572be938d6b6c-objects.json: ReferenceError: options is not defined [2024-10-14 08:24:52.091]: storage.purge warn Could not remove orphan file ff/5f/9a8df41d85f158a33f78c6d572be938d6b6c-similarity-embeddings.json: ReferenceError: options is not defined [2024-10-14 08:24:52.092]: storage.purge warn Could not remove orphan file ff/5f/9a8df41d85f158a33f78c6d572be938d6b6c-vibrant.json: ReferenceError: options is not defined [2024-10-14 08:24:52.094]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-exif.json: ReferenceError: options is not defined [2024-10-14 08:24:52.095]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-faces.json: ReferenceError: options is not defined [2024-10-14 08:24:52.095]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-image-preview-128.jpg: ReferenceError: options is not defined [2024-10-14 08:24:52.096]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-image-preview-1280.jpg: ReferenceError: options is not defined [2024-10-14 08:24:52.097]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-image-preview-320.jpg: ReferenceError: options is not defined [2024-10-14 08:24:52.098]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-image-preview-800.jpg: ReferenceError: options is not defined [2024-10-14 08:24:52.099]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-objects.json: ReferenceError: options is not defined [2024-10-14 08:24:52.100]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-similarity-embeddings.json: ReferenceError: options is not defined [2024-10-14 08:24:52.101]: storage.purge warn Could not remove orphan file ff/74/2f8ede8259ec97b887daaa08535fdae6660a-vibrant.json: ReferenceError: options is not defined

I am running the current version (1.18.0) of the docker image via docker-compose as described in the documentation.

[xemle]

Hi. Thank you for your time and issue. I wont be able to answer within a short time and plan to answer within the next 7 days.

[EinAeffchen]

I'm not very experienced with js per se, but if you have a hint for me regarding where to look, I might give it a shot myself.

[xemle]

Thank you for the poke. I am sorry that I was not able yet to answer. Usually it is quite fun and relaxing to working on the gallery topic but somehow the break became long and I need some time to step in again.

I plan to step in again but was not able yet. Pleas be patient.

In the meanwhile you can check the json logs via jq. Usually errors are logged with stack traces a gives you a hint where the invalid assignment is located.

[xemle]

Hi @EinAeffchen

I found some time and fixed the issue in a290f75. Basically it was an issue with the logging of file removal.

I recently removed some files from my library and noticed they kept showing up on the website anyways

After you remove files from you library the database must be rebuild. This is done via 3 steps:

• updating the file index
• extract meta data and calculate preview
• build the database

By default a file watcher is running and does this job for you.

The file index will discover that files are removed and further processes will remove these files from your database. The storage will keep the files for performance reasons.

if I understand correctly that is supposed to remove orphaned images right?

Correct. It removes all orphaned extractor files like previews or exif data from the storage.

Due to the fact that (self hosted) storage is cheap, this is an optional step which I do not run. But if you have lots of removed data and you like to cleanup your storage this is the way to go.

When I run the command it fails to delete anything due to a bug though:

According to the code the bug targets only the success message logging of removed file and the orphan files are removed successful before.

So the question is: Why you have removed files and they are still in you gallery while the standard file watcher should do the job? Have customize something special? The gallery will update the files after a docker container restart. Are the files still there?

[EinAeffchen]

Hi Xemle, thanks for looking into this!

So I actually completely removed all original content and readded it with lower quality and changed signature.

The file index will discover that files are removed and further processes will remove these files from your database. The storage will keep the files for performance reasons.

Is the file index maybe related to the file names? Because I reuploaded the changed images under the same filenames, that would explain why the removal wasn't discovered and the thumbnails etc. are still there.

I just updated and restarted the container, but I'm still seeing the versions of the original images. Even marking them with tags and deleting them with the gallery.js database remove -q tag:trash command doesn't seem to remove anything for me, despite the command running successfully.

[xemle]

Hi @EinAeffchen

The media id id derived from the content and it is a plain SHA1 checksum. If your new insert files are not changed in a byte but renamed or moved, it will lead to the same id and therefore it wont be detected as orphan storage file. The reason for this is that I want to support file moves/renames/reorganization of the folder structure without recalculating all the previews which are time expensive. So when ever I like to reorganize my folders the rebuild of the gallery does not require lots of time.

Further gallery.js database remove -q tag:trash will remove files in the first run from the database but on the next import the files will be added again to the gallery. The command database remove is meant to be used if you import media from a remote gallery and you like to remove them on your local gallery.

Maybe it is worth to have a look to the documentation of the internal building blocks and design decisions to understand how the gallery is working.

Currently I have the feeling that I do not get your problem or your task which you like to solve. Maybe we can have a chat conversation on discord to clarify things?