[BUG] Web UI does not delete media from server / Cannot be reuploaded

[andrewsris]

The bug

I am currently migrating my library of photos to immich. After uploading to immich, I have deleted a number of photos that were categorized on the wrong date due to incorrect metadata / upload bugs. However, the Immich web interface did not delete the photo from my server. Thus, when I try to reupload the photos in bulk using the CLI, it fails to upload any these files because they already exist on the server, though I cannot actually see them in the interface.

This is particularly challenging with large libraries that I was manually cleaning up. In large libraries, it will be challenging to detect if photos are uploaded and visible in the interface or not.

The OS that Immich Server is running on

Unraid 6.11.1

Version of Immich Server

v1.50.1

Version of Immich Mobile App

N/A

Platform with the issue

• [X] Server
• [X] Web
• [ ] Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: altran1502/immich-server:release
    entrypoint: [ "/bin/sh", "./start-server.sh" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: altran1502/immich-server:release
    entrypoint: [ "/bin/sh", "./start-microservices.sh" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: altran1502/immich-machine-learning:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    restart: always

  immich-web:
    container_name: immich_web
    image: altran1502/immich-web:release
    entrypoint: [ "/bin/sh", "./entrypoint.sh" ]
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: altran1502/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/mnt/user/photos/immich

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

Reproduction steps

1. Upload photos using immich CLI
2. Within the webUI, delete photos
3. Re-upload photos using immich CLI

Additional information

I have used this command from the immich CLI docker run -it --rm -v "$(pwd)":/import ghcr.io/immich-app/immich-cli:latest upload --key ############### --server http://192.168.1.100:2283/api

I am attaching an image of the web UI showing the discrepancy of what can be seen (no photos) and what it detects (278 videos and 92 photos). I am also attaching an image of what is left on the server for this immich user, which you can see is not empty.

[alextran1502]

Currently, I need help reproducing the issue because I can't seem to replicate this bug on my instance. Can you help me follow the step below

1. Create a new instance
2. From your computer, add a few pictures into that directory where you can keep track of the number
3. Upload from the CLI and count the number of files that get uploaded at the target location
4. Remove photos/videos from the web interface
5. Recount the files number at the target location
6. Reupload using the CLI

Let me know if you can reproduce the issue

[andrewsris]

Thanks. Just to clarify, by "instance," do you mean to create a new user and try the whole process again? If so, then interestingly I cannot recreate the EXACT issue, but I am finding a whole number of new issues that are likely related.

First, I created a new user and uploaded a sample folder of 5 .jpg photos that I was able to upload (using CLI) and delete (from web interface) back and forth without any issue.

Second, with this new user, I tried to upload the folder of 98 media files (67 mp4, 28 jpg, and 3 png) that gave me the problem in the first place. Surprisingly I was able to upload and delete these without the main issue that caused me to open this thread. However, I found a number of new issues. Notably that only 93 / 98 files are shown in the interface, and that immich double counts all the files for some reason (see first attached image). After trying to delete all of the media from the web interface, the remaining files in the directory are all of the mp4s (both in the specific date and "encoded-video" folder) and many .jpg or .webp files in the "thumb/CLI" folder.

Third, I tried to go back to the user that started this issue. I tried to re-upload this problematic folder of 98 media files, and just like the first post, the immich CLI will not let me upload any file. Looking into the file directory of this problematic user, it seems like Immich created a second file with a "+1" addended into the file name (see second attached image).

Fourth, I looked into the problematic media folder again, which is on a USB drive that I created on a MacBook. Unknown to me, it seems like MacOS created hidden files with names ._filename (see third attached image). Do you think the issues could either be from the mp4 files or the hidden "._" files?

[alextran1502]

The problem are the hidden mp4 files. I am quite certain

[andrewsris]

Thanks! I can confirm that the hidden files were the issue. After deleting the hidden files from my USB, I can now use the immich CLI to upload and the Web interface to delete freely on all of my media files (with the caveat that the encoded-videos folder still doesn't get deleted).

I'm glad to see where the problem lies, but for my main library (~27,000 files), this will be challenging to troubleshoot the damage already done by the hidden files. Maybe I should just start over?

[alextran1502]

Yes, I would suggest starting over.