[BUG] Web/CLI upload successful, but some images are missing

[rootiam]

The bug

I am uploading several images to my immich server and some are always missing. It doesn't matter if I use the Web Upload or the bulk CLI one. Some are uploaded fine and some others not. Both web and cli are telling the upload was successful. I can also repeat the upload. The images are still missing. I tried another user and there the images are uploaded fine. So I guess it is some kind of database error or something?

The OS that Immich Server is running on

Ubuntu 22.04.2 LTS

Version of Immich Server

1.60.0

Version of Immich Mobile App

1.60.0 build.83

Platform with the issue

• [X] Server
• [X] Web
• [ ] Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich-server
    image: altran1502/immich-server:release
    command: ["start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redisimmich
      - dbimmich
      - typesense
    restart: unless-stopped
    networks:
      - proxy
    labels:
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"
# Homepage Integration
      - homepage.group=Selfhosted 1
      - homepage.name=Immich
#      - homepage.icon=emby.png
      - homepage.href=https://photos.xxx.xx
      - homepage.description=Photos

  immich-microservices:
    container_name: immich-microservices
    image: altran1502/immich-server:release
    command: ["start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
      - TZ=Europe/Berlin
    depends_on:
      - redisimmich
      - dbimmich
      - typesense
    restart: unless-stopped
    networks:
      - proxy
    labels:
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"

  immich-machine-learning:
    container_name: immich-machine-learning
    image: altran1502/immich-machine-learning:release
    command: ["python", "src/main.py"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - dbimmich
    restart: unless-stopped
    networks:
      - proxy
    labels:
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"

  immich-web:
    container_name: immich-web
    image: altran1502/immich-web:release
#    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - .env
    restart: unless-stopped
    networks:
      - proxy
    labels:
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"

  redisimmich:
    container_name: immich-redis
    image: redis:6.2
    restart: unless-stopped
    networks:
      - proxy
    labels:
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"

 dbimmich:
    container_name: immich-postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - ./immich_db_data:/var/lib/postgresql/data
    restart: unless-stopped
    networks:
      - proxy
    labels:
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"

  typesense:
    container_name: immich-typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    volumes:
      - ./immich_typesense_data:/data
    networks:
      - proxy
    env_file:
      - .env
    restart: unless-stopped
    labels:
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"

  immich-proxy:
    container_name: immich-proxy
    image: altran1502/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
#    logging:
#      driver: none
    depends_on:
      - immich-server
    restart: unless-stopped
    labels:
      - "traefik.http.routers.immich.rule=Host(`photos.xxx.xx`)"
      - "traefik.http.routers.immich.entrypoints=websecure"
      - "traefik.http.routers.immich.tls=true"
      - "traefik.http.routers.immich.tls.certresolver=mydnschallenge"
      # Watchtower Update
      - "com.centurylinklabs.watchtower.enable=true"
    networks:
      - proxy

networks:
  proxy:
    external: true
volumes:
  model-cache:

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich-postgres
DB_USERNAME=xxxx
DB_PASSWORD=xxxxx
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich-redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=./immich_upload

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

DISABLE_REVERSE_GEOCODING=false
REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

TYPESENSE_API_KEY=xxxx

Reproduction steps

1. upload images via web or bulk cli
2. upload was successful 
3. some images are missing 
4. retry upload
5. images are still missing
...

Additional information

No response

[alextran1502]

Please check the server/microservices logs. Can you also run the thumbnail generation job in the admin panel as well?

[rootiam]

I checked the logs, then started the upload again via cli and checked the logs after the upload again. Neither in the server logs nor in the microservices logs were any logs added. I also checked the logs in general and cannot see anything weird.

I triggered the thumbnail generation for all files. Will take some time. I will report back.

I just also checked the filesystem. The files are missing there too.

[alextran1502]

If the files are missing, it is either not supported or duplicated

[rootiam]

It must be supported. I tried with another user and it worked perfectly.

If it is duplicated. Where are those files then? How could I debug or clean it?

[jrasm91]

Can you upload via the web and inspect the network request for the upload? The response should indicate why it would be rejected.

[rootiam]

@jrasm91 I tried that. HTTP response 200 for the uploaded files. But still missing.

@alextran1502 Thumbnail generation task finished over night. The images are still missing.

I noticed another thing, not sure if related. The images are in the date range 21st to 28th of July 2018. Those are all missing. The images in the gallery are up to 15th of July 2018 and the next ones are starting 12th of August 2018. When scrolling in the Web over this timeframe the scrolling is always jumping! Otherwise it is always smooth. I can reproduce this every time when scrolling to this section. Maybe Web is trying to load pictures and fails and that's why it's jumping?

[jrasm91]

200 response means the the file already exists on the server (duplicate). 201 would indicate a new record was created. If you click on the request and look at the response you would see { duplicate: true, id: <UUID> }

[rootiam]

Yes that is correct. After some more searching I found the files. They were sorted to April 2021 (along with some other files from earlier years). There must have been some issues during past copy jobs or something so the metadata wasn't correct anymore. With this I can close the issue. Thanks guys, awesome work!