[BUG] Interrupted mobile uploads leave corrupt files

The bug

Every few thousand uploads something goes wrong and the upload is stopped mid file. This results in errors during file processing. As a result these uploads are stuck on the untracked files section of the repair tab.

There are a few issues with this:

Apparently partial uploads are not detected by the upload code. Given that some partial uploads might be able to get processed as if they were a correct file this means there is no way to know if a part of the library is corrupt.
The files that are partial enough to get stuck in the untracked files sections are stuck there. There is some (older?) documentation referring to a "Remove Offline Files" job that does not seem to exist for the default user library?
Even though the server knows some files can't be processed (if your are lucky). The mobile app incorrectly shows everything is just fine while it's not.

The most egregious issue to me seems to be issue number 1. Especially for broken uploads that don't get detected as corrupt. A relative easy fix for this would be to upload from the app not only the file, but also the checksum. And then only accept the file server side if the checksum checks out. If not, drop the upload and ask the app to try again.

Update: Seems @ItalyPaleAle already ran into this wall once before. Not sure why https://github.com/immich-app/immich/issues/4532 was closed.

The OS that Immich Server is running on

Unraid v6.12.10

Version of Immich Server

v1.105.1

Version of Immich Mobile App

v1.105.0

Platform with the issue

[X] Server
[ ] Web
[X] Mobile

Your docker-compose.yml content

# backup stack

networks:
  default:
    name: backup
services:

  immich:
    container_name: backup-immich
    image: ghcr.io/immich-app/immich-server:release
    command: ['start.sh', 'immich']
    restart: always
    depends_on:
      - redis-immich
      - postgres-immich
    volumes:
      - /mnt/user/immich:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    environment:
      TZ: Europe/Amsterdam
      PUID: 99
      PGID: 100
      REDIS_HOSTNAME: redis-immich
      DB_HOSTNAME: postgres-immich
      DB_DATABASE_NAME: immich
      DB_USERNAME: postgres
      DB_PASSWORD: REDACTED
    labels:
      traefik.enable: true
      traefik.http.services.immich-backup.loadbalancer.server.port: 3001

  microservices-immich:
    container_name: backup-immich-microservices
    image: ghcr.io/immich-app/immich-server:release
    command: ['start.sh', 'microservices']
    restart: always
    depends_on:
      - redis-immich
      - postgres-immich
    volumes:
      - /mnt/user/immich:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["GPU-e1474488-7fa9-85a5-803b-59c645d71e0d"]
              capabilities: [gpu]
    environment:
      TZ: Europe/Amsterdam
      PUID: 99
      PGID: 100
      NVIDIA_DRIVER_CAPABILITIES: all
      NVIDIA_VISIBLE_DEVICES: all
      REDIS_HOSTNAME: redis-immich
      DB_HOSTNAME: postgres-immich
      DB_DATABASE_NAME: immich
      DB_USERNAME: postgres
      DB_PASSWORD: REDACTED

  machinelearning-immich:
    container_name: backup-immich-machinelearning
    image: ghcr.io/immich-app/immich-machine-learning:release-cuda
    restart: always
    volumes:
      - /mnt/user/cache/backup/immich/machinelearning:/cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["GPU-e1474488-7fa9-85a5-803b-59c645d71e0d"]
              capabilities: [gpu]
    environment:
      TZ: Europe/Amsterdam
      PUID: 99
      PGID: 100
      NVIDIA_DRIVER_CAPABILITIES: all
      NVIDIA_VISIBLE_DEVICES: all

  redis-immich:
    container_name: backup-immich-redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always
    volumes:
      - /mnt/user/cache/backup/immich/redis:/data
    environment:
      TZ: Europe/Amsterdam
      PUID: 99
      PGID: 100

  postgres-immich:
    container_name: backup-immich-postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always
    volumes:
      - /mnt/user/containers/backup/immich-postgres:/var/lib/postgresql/data
    environment:
      TZ: Europe/Amsterdam
      PUID: 99
      PGID: 100
      POSTGRES_INITDB_ARGS: '--data-checksums'
      POSTGRES_DB: immich
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: REDACTED

Your .env content

In-lined into the compose file.

Reproduction steps

Attempt mobile uploads on a large number of files and mess with the connection by:

- Rapidly changing between foreground and background settings during the upload.
- Switch between WiFi and mobile networks.
- Kill and restart the app a few times.

Then check the server repair page. There will be some files.

Relevant log output

Can't find the relevant log anymore. But it was something generic about not being able to read the file. This is expected as the file has only been partially uploaded.

Additional information

This is a (intentionally very low res) screenshot of the two files compared. Left the uploaded file. Right the original file. You can clearly see how the file size of about 10% is reflected in the image.

immich-app / immich