search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
53.44k stars 2.83k forks source link

Not uploading all photos? #2380

Closed wouterverduin closed 1 year ago

wouterverduin commented 1 year ago

The bug

I want to use immich as a nice backup to get the files from my phone to my server. Installed using dock-ercompose and set up the app on my phone. After logging in and selecting the 'camera' folder it starts uploading (~2500 images). After some time it is done (although it says as remaing -78 photos?) However it doenst upload ALL the images. Most images in the app show a cload with a checking mark, however quite some have a cloud with a slash through it.

How do i make the app upload ALL images? Am i missing a setting or is there some other reason? I cant find errors in the logs either. It just seems to randomly decides which to upload and which not.

The OS that Immich Server is running on

Ubuntu 20.04; Docker

Version of Immich Server

V1.54.1

Version of Immich Mobile App

v1.54.0 build 77

Platform with the issue

Your docker-compose.yml content

version: "3.3"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - /MEDIA/Immich:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - /MEDIA/Immich:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - /MEDIA/Immich:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - ./tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always
  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: 
      POSTGRES_USER: 
      POSTGRES_DB: 
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - ./pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2253:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=
DB_USERNAME=
DB_PASSWORD=
DB_DATABASE_NAME=

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/MEDIA/Immich

###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=some-random-text
# TYPESENSE_ENABLED=false

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################

Reproduction steps

1. Docker-compose up -d
2. Select the 'camera' folder in the app
3. Wait and see random selection of missing photos

Removed all settings and files and did a fresh restart without succes so far...

Additional information

No response

samip5 commented 1 year ago

Which mobile platform? iOS or Android?

wouterverduin commented 1 year ago

Android!

alextran1502 commented 1 year ago

Hello, can you export the log in the mobile app and attach it here? Thank you

wouterverduin commented 1 year ago

https://drive.google.com/file/d/1hby-oIreZagLQ3zvV0QXxMrq6IY8W0Jf/view?usp=sharing

alextran1502 commented 1 year ago

Hmm I don't see anything strange in the log. If you pull to refresh do you see the icon changes? Do you see anything in the server or microservices container log?

maliqu commented 1 year ago

I have a similar problem, everything worked fine, until some days off. I disconnected my Android mobile from the network for a few days. Now the app cannot synchronize photos from these days, only the new ones (after running v1.54.0). v1.54.1 not syncing at all. From the very beginning, I selected Recent (all) and Camera as a source of photos and videos. (The same situation on my wife's phone)

Sometimes I see something like that (server)

...
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId/users, PUT} route +2ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId/assets, PUT} route +1ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId, GET} route +1ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId/assets, DELETE} route +1ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId, DELETE} route +0ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId/user/:userId, DELETE} route +1ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId, PATCH} route +1ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/:albumId/download, GET} route +0ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [RouterExplorer] Mapped {/album/create-shared-link, POST} route +1ms
[Nest] 1  - 05/05/2023, 11:46:13 PM     LOG [SearchService] Running bootstrap
Request #1683330373399: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1683330373399: Sleeping for 4s and then retrying request...
[Nest] 1  - 05/05/2023, 11:46:17 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v5
[Nest] 1  - 05/05/2023, 11:46:17 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v1
[Nest] 1  - 05/05/2023, 11:46:17 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"albums-v1","name":"albums"},{"collection_name":"assets-v5","name":"assets"}]
[Nest] 1  - 05/05/2023, 11:46:17 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false}
[Nest] 1  - 05/05/2023, 11:46:17 PM     LOG [NestApplication] Nest application successfully started +15ms
[Nest] 1  - 05/05/2023, 11:46:17 PM     LOG [ImmichServer] Running Immich Server in PRODUCTION environment - version 1.54.0 - Listening on port: 3001
[Nest] 1  - 05/05/2023, 11:46:17 PM    WARN [ImmichServer] Machine learning is enabled
[Nest] 1  - 05/05/2023, 11:46:17 PM    WARN [ImmichServer] Search is enabled
[Nest] 1  - 05/05/2023, 11:46:21 PM     LOG [CommunicationGateway] New websocket connection: 5G_Mo_-7O56mAMrHAAAB

and something like that (microservices)

...
[Nest] 1  - 05/05/2023, 11:31:57 PM    WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
[Nest] 1  - 05/05/2023, 11:31:57 PM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
[Nest] 1  - 05/05/2023, 11:31:57 PM     LOG [InstanceLoader] InfraModule dependencies initialized +54ms
[Nest] 1  - 05/05/2023, 11:31:57 PM     LOG [InstanceLoader] DomainModule dependencies initialized +1ms
[Nest] 1  - 05/05/2023, 11:31:57 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 1  - 05/05/2023, 11:31:57 PM     LOG [NestApplication] Nest application successfully started +36ms
[Nest] 1  - 05/05/2023, 11:31:57 PM     LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.54.0 - Listening on port: 3002
[Nest] 1  - 05/05/2023, 11:32:27 PM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized
[Nest] 1  - 05/05/2023, 11:35:06 PM   ERROR [SmartInfoService] Unable run clip encoding pipeline: 27b21f61-0f53-4a53-b82c-1b4f12a026f6
Error: socket hang up
    at connResetException (node:internal/errors:692:14)
    at Socket.socketOnEnd (node:_http_client:478:23)
    at Socket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [NestFactory] Starting Nest application...
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +141ms
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +2ms
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +15ms
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 1  - 05/05/2023, 11:46:11 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 1  - 05/05/2023, 11:46:12 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +363ms
[Nest] 1  - 05/05/2023, 11:46:12 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +2ms
[Nest] 1  - 05/05/2023, 11:46:12 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 1  - 05/05/2023, 11:46:12 PM    WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
[Nest] 1  - 05/05/2023, 11:46:12 PM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
[Nest] 1  - 05/05/2023, 11:46:31 PM     LOG [InstanceLoader] InfraModule dependencies initialized +19688ms
[Nest] 1  - 05/05/2023, 11:46:31 PM     LOG [InstanceLoader] DomainModule dependencies initialized +0ms
[Nest] 1  - 05/05/2023, 11:46:31 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms
[Nest] 1  - 05/05/2023, 11:46:32 PM     LOG [NestApplication] Nest application successfully started +34ms
[Nest] 1  - 05/05/2023, 11:46:32 PM     LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.54.0 - Listening on port: 3002
[Nest] 1  - 05/05/2023, 11:46:32 PM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized

Is it possible to add a button in the mobile app to force upload (backup) selected assets?

alextran1502 commented 1 year ago

@maliqu It seems like there is some bugs in the local synchronization mechanism that cause this issue under some specific conditions that we have not fully understood yet. If you try to reinstall the app, does it help?

bozzfozz commented 1 year ago

I have to force quit via menu on android app and then it works again

wouterverduin commented 1 year ago

I tried both force quitting the app and reponening it and reinstalling it without succes.

Here is a grasp of the last few days of the immich_server docker logs (made photos on the sixth of may the latest):

[Nest] 1 - 05/06/2023, 8:27:27 AM LOG [CommunicationGateway] New websocket connection: qlR72tzYX8YmhhhXAAD5 [Nest] 1 - 05/06/2023, 8:27:41 AM LOG [CommunicationGateway] Client qlR72tzYX8YmhhhXAAD5 disconnected from Websocket [Nest] 1 - 05/06/2023, 8:28:16 AM LOG [CommunicationGateway] New websocket connection: XQ0w9h9rCYTDSVexAAD7 [Nest] 1 - 05/06/2023, 8:28:20 AM LOG [CommunicationGateway] Client XQ0w9h9rCYTDSVexAAD7 disconnected from Websocket [Nest] 1 - 05/06/2023, 8:28:27 AM LOG [CommunicationGateway] New websocket connection: Cg4jXFy5fEusy7P7AAD9 [Nest] 1 - 05/06/2023, 8:28:32 AM LOG [CommunicationGateway] Client Cg4jXFy5fEusy7P7AAD9 disconnected from Websocket [Nest] 1 - 05/06/2023, 8:28:38 AM LOG [CommunicationGateway] New websocket connection: FSoc8JWSXDvOTy9iAAD [Nest] 1 - 05/06/2023, 8:28:40 AM LOG [CommunicationGateway] Client FSoc8JWSXDvOTy9iAAD disconnected from Websocket [Nest] 1 - 05/06/2023, 8:30:12 AM LOG [CommunicationGateway] New websocket connection: tLTA8xAWPDxtFyOBAAEB [Nest] 1 - 05/06/2023, 8:30:31 AM LOG [CommunicationGateway] Client tLTA8xAWPDxtFyOBAAEB disconnected from Websocket [Nest] 1 - 05/06/2023, 8:42:46 AM LOG [CommunicationGateway] New websocket connection: i8vIc4TCabbBg41kAAED [Nest] 1 - 05/06/2023, 8:43:00 AM LOG [CommunicationGateway] Client i8vIc4TCabbBg41kAAED disconnected from Websocket [Nest] 1 - 05/06/2023, 8:44:01 AM LOG [CommunicationGateway] New websocket connection: XidP-lQ4Qb2UIrNsAAEF [Nest] 1 - 05/06/2023, 8:44:03 AM LOG [CommunicationGateway] Client XidP-lQ4Qb2UIrNsAAEF disconnected from Websocket [Nest] 1 - 05/06/2023, 9:02:45 AM LOG [CommunicationGateway] New websocket connection: mwFusSRRi-JiMogIAAEH [Nest] 1 - 05/06/2023, 9:02:52 AM LOG [CommunicationGateway] Client mwFusSRRi-JiMogIAAEH disconnected from Websocket [Nest] 1 - 05/06/2023, 9:03:31 AM LOG [CommunicationGateway] New websocket connection: QeQVRLmtV9slitntAAEJ [Nest] 1 - 05/06/2023, 9:03:31 AM LOG [CommunicationGateway] New websocket connection: TJh5eZBk1EVji81XAAEL [Nest] 1 - 05/06/2023, 9:03:32 AM LOG [CommunicationGateway] New websocket connection: 68wNf0GPzJQmRWvdAAEN [Nest] 1 - 05/06/2023, 9:03:41 AM LOG [CommunicationGateway] Client 68wNf0GPzJQmRWvdAAEN disconnected from Websocket [Nest] 1 - 05/06/2023, 9:05:40 AM LOG [CommunicationGateway] Client QeQVRLmtV9slitntAAEJ disconnected from Websocket [Nest] 1 - 05/06/2023, 9:05:40 AM LOG [CommunicationGateway] Client TJh5eZBk1EVji81XAAEL disconnected from Websocket [Nest] 1 - 05/06/2023, 9:09:27 AM LOG [CommunicationGateway] New websocket connection: 1_93BQ2SjzGRR6kgAAER [Nest] 1 - 05/06/2023, 9:09:27 AM LOG [CommunicationGateway] New websocket connection: ZpZvr4Knr9M7T5fZAAES [Nest] 1 - 05/06/2023, 9:09:27 AM LOG [CommunicationGateway] New websocket connection: nEF9GbFoozmOWe0lAAET [Nest] 1 - 05/06/2023, 9:09:29 AM LOG [CommunicationGateway] Client nEF9GbFoozmOWe0lAAET disconnected from Websocket [Nest] 1 - 05/06/2023, 9:11:59 AM LOG [CommunicationGateway] Client 1_93BQ2SjzGRR6kgAAER disconnected from Websocket [Nest] 1 - 05/06/2023, 9:11:59 AM LOG [CommunicationGateway] Client ZpZvr4Knr9M7T5fZAAES disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:00 AM LOG [CommunicationGateway] New websocket connection: fiQkaTZ_hs5IsS9PAAEW [Nest] 1 - 05/06/2023, 9:49:00 AM LOG [CommunicationGateway] New websocket connection: i-69PnXbg1oL-B2QAAEX [Nest] 1 - 05/06/2023, 9:49:09 AM LOG [CommunicationGateway] Client fiQkaTZ_hs5IsS9PAAEW disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:09 AM LOG [CommunicationGateway] Client i-69PnXbg1oL-B2QAAEX disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:41 AM LOG [CommunicationGateway] New websocket connection: V5Pu_jDUxGV1x8EjAAEZ [Nest] 1 - 05/06/2023, 9:49:42 AM LOG [CommunicationGateway] New websocket connection: dnylbhToTz-3LJtqAAEb [Nest] 1 - 05/06/2023, 9:49:42 AM LOG [CommunicationGateway] New websocket connection: yMy_ElpP_hzAIYMBAAEd [Nest] 1 - 05/06/2023, 9:49:51 AM LOG [CommunicationGateway] Client yMy_ElpP_hzAIYMBAAEd disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:52 AM LOG [CommunicationGateway] Client dnylbhToTz-3LJtqAAEb disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:52 AM LOG [CommunicationGateway] Client V5Pu_jDUxGV1x8EjAAEZ disconnected from Websocket [Nest] 1 - 05/08/2023, 5:21:32 AM LOG [CommunicationGateway] New websocket connection: LwpGouSHwL5HL-yIAAEf [Nest] 1 - 05/08/2023, 5:21:37 AM LOG [CommunicationGateway] Client LwpGouSHwL5HL-yIAAEf disconnected from Websocket [Nest] 1 - 05/08/2023, 5:21:52 AM LOG [CommunicationGateway] New websocket connection: 5ujGEioyrElFQC40AAEh [Nest] 1 - 05/08/2023, 5:21:52 AM LOG [CommunicationGateway] Client 5ujGEioyrElFQC40AAEh disconnected from Websocket [Nest] 1 - 05/08/2023, 5:21:55 AM LOG [CommunicationGateway] New websocket connection: nEewhxlm7vpVWUlgAAEj [Nest] 1 - 05/08/2023, 5:22:08 AM LOG [CommunicationGateway] Client nEewhxlm7vpVWUlgAAEj disconnected from Websocket

This is the last part of the immich_microservices logs; Not sure how to interpret them myself but atleast shows errors:

Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/77648712-fce4-48a8-b033-9a0852d473fb.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/05/2023, 8:09:20 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=bc5dc290-3f95-4095-b0e2-54418ade18dd) [Nest] 1 - 05/05/2023, 8:09:20 AM ERROR [MediaService] Failed to generate thumbnail for asset: bc5dc290-3f95-4095-b0e2-54418ade18dd/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/e0d66649-6087-47c1-af95-5d0b11dd7b21.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) at runMicrotasks () at runNextTicks (node:internal/process/task_queues:61:5) at processTimers (node:internal/timers:499:9) [Nest] 1 - 05/05/2023, 8:09:21 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=133cde0c-245c-4054-9307-370bf17464dc) [Nest] 1 - 05/05/2023, 8:09:21 AM ERROR [MediaService] Failed to generate thumbnail for asset: 133cde0c-245c-4054-9307-370bf17464dc/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/c9a31515-f9c9-4101-8c59-c0677b438783.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) at runMicrotasks () at runNextTicks (node:internal/process/task_queues:61:5) at processTimers (node:internal/timers:499:9) [Nest] 1 - 05/05/2023, 8:09:21 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=b3ee5bd4-d80c-45fd-9557-27ac6652aed2) [Nest] 1 - 05/05/2023, 8:09:21 AM ERROR [MediaService] Failed to generate thumbnail for asset: b3ee5bd4-d80c-45fd-9557-27ac6652aed2/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/93f1a63e-d3f0-4e89-9570-cb5dd44557cf.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/05/2023, 8:09:21 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=4addc73c-e353-4445-9fd6-f7b9c5b971b7) [Nest] 1 - 05/05/2023, 8:09:22 AM ERROR [MediaService] Failed to generate thumbnail for asset: 4addc73c-e353-4445-9fd6-f7b9c5b971b7/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/e9796d25-dc58-4937-be74-11bf2da323d7.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/05/2023, 8:09:22 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=fa1c65f2-35c9-4bb8-8344-7079a7c405ec) [Nest] 1 - 05/05/2023, 8:09:22 AM ERROR [MediaService] Failed to generate thumbnail for asset: fa1c65f2-35c9-4bb8-8344-7079a7c405ec/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/eecfa193-c038-4ccc-91e3-d56f4313a918.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/05/2023, 11:54:57 PM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=ecb5b0fc-b555-47b9-8311-302ef3a4dd96) [Nest] 1 - 05/05/2023, 11:54:58 PM ERROR [MediaService] Failed to generate thumbnail for asset: ecb5b0fc-b555-47b9-8311-302ef3a4dd96/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/72047b5a-ae5b-4a6f-9b5f-36349cc57a54.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/06/2023, 9:58:51 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=8ef5ca99-3253-4a13-9010-3be844ae51d6) [Nest] 1 - 05/06/2023, 9:58:51 AM ERROR [MediaService] Failed to generate thumbnail for asset: 8ef5ca99-3253-4a13-9010-3be844ae51d6/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/cdd0e244-5ecb-4d91-882b-8de14a7bdf42.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/06/2023, 9:58:52 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=88e1caae-6ae2-45db-b720-d10589118f26) [Nest] 1 - 05/06/2023, 9:58:52 AM ERROR [MediaService] Failed to generate thumbnail for asset: 88e1caae-6ae2-45db-b720-d10589118f26/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/556740c9-c7d4-4bae-9265-7611d48d31ef.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)

Any clue how to proceed?

alextran1502 commented 1 year ago

@wouterverduin As you see in the microservices log, there are some thumbnail generation job has error because of file not found. Can you exec into the microservice container and try to look for a file that is logged as not found in one of the error message?

Can you also run the missing thumbnail generation job in the admin panel on the web and monitor the server/microservices log as well?

wouterverduin commented 1 year ago

@alextran1502

I additionaly found this error along in the microservices:

Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/99a79ba1-caac-447a-bc84-e28d2fc77589.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) at runMicrotasks (<anonymous>) at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7)

Then i looked at the last two entries of the microservices log: [Nest] 1 - 05/06/2023, 9:58:51 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=8ef5ca99-3253-4a13-9010-3be844ae51d6) [Nest] 1 - 05/06/2023, 9:58:51 AM ERROR [MediaService] Failed to generate thumbnail for asset: 8ef5ca99-3253-4a13-9010-3be844ae51d6/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/cdd0e244-5ecb-4d91-882b-8de14a7bdf42.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/06/2023, 9:58:52 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=88e1caae-6ae2-45db-b720-d10589118f26) [Nest] 1 - 05/06/2023, 9:58:52 AM ERROR [MediaService] Failed to generate thumbnail for asset: 88e1caae-6ae2-45db-b720-d10589118f26/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/556740c9-c7d4-4bae-9265-7611d48d31ef.jpg

I somehow couldnt use /bin/bash or bash to get into the container? Nonetheless ran the ls command to look into the folder and see if i could find the file "556740c9-c7d4-4bae-9265-7611d48d31ef.jpg" or "cdd0e244-5ecb-4d91-882b-8de14a7bdf42.jpg" Both files are not in that folder. And strangely there are only about 20 files in that folder despite my image folder being about 2500. I also checked the folder since i have it mapped as a network share aswell. What struck me was that i have 4 folders in the folder i pointed my immich installation too (encoded-video/library/thumbs/upload). The last one (upload) was corresponding to the folder the error logs where point to considering it had the same 20 files in it. Both the libary and upload folder had the "f079c46c-3b61-4541-9979-1ebb3f8b07ad" folder in it. In the library one it was succeeded by folders corresponding to years. Both the files mentioned above wherent findable in the library folder.

Secondly, the missing thumbnail generation job:

Starting the job it says active 1 and then changes to 7. And in the waiting panel it shows 1223

The microservices last entries remain the same before and after the task:

Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/c9a31515-f9c9-4101-8c59-c0677b438783.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) at runMicrotasks (<anonymous>) at runNextTicks (node:internal/process/task_queues:61:5) at processTimers (node:internal/timers:499:9) [Nest] 1 - 05/05/2023, 8:09:21 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=b3ee5bd4-d80c-45fd-9557-27ac6652aed2) [Nest] 1 - 05/05/2023, 8:09:21 AM ERROR [MediaService] Failed to generate thumbnail for asset: b3ee5bd4-d80c-45fd-9557-27ac6652aed2/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/93f1a63e-d3f0-4e89-9570-cb5dd44557cf.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/05/2023, 8:09:21 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=4addc73c-e353-4445-9fd6-f7b9c5b971b7) [Nest] 1 - 05/05/2023, 8:09:22 AM ERROR [MediaService] Failed to generate thumbnail for asset: 4addc73c-e353-4445-9fd6-f7b9c5b971b7/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/e9796d25-dc58-4937-be74-11bf2da323d7.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/05/2023, 8:09:22 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=fa1c65f2-35c9-4bb8-8344-7079a7c405ec) [Nest] 1 - 05/05/2023, 8:09:22 AM ERROR [MediaService] Failed to generate thumbnail for asset: fa1c65f2-35c9-4bb8-8344-7079a7c405ec/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/eecfa193-c038-4ccc-91e3-d56f4313a918.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/05/2023, 11:54:57 PM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=ecb5b0fc-b555-47b9-8311-302ef3a4dd96) [Nest] 1 - 05/05/2023, 11:54:58 PM ERROR [MediaService] Failed to generate thumbnail for asset: ecb5b0fc-b555-47b9-8311-302ef3a4dd96/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/72047b5a-ae5b-4a6f-9b5f-36349cc57a54.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/06/2023, 9:58:51 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=8ef5ca99-3253-4a13-9010-3be844ae51d6) [Nest] 1 - 05/06/2023, 9:58:51 AM ERROR [MediaService] Failed to generate thumbnail for asset: 8ef5ca99-3253-4a13-9010-3be844ae51d6/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/cdd0e244-5ecb-4d91-882b-8de14a7bdf42.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) [Nest] 1 - 05/06/2023, 9:58:52 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=88e1caae-6ae2-45db-b720-d10589118f26) [Nest] 1 - 05/06/2023, 9:58:52 AM ERROR [MediaService] Failed to generate thumbnail for asset: 88e1caae-6ae2-45db-b720-d10589118f26/IMAGE Error: File not found - /usr/src/app/upload/upload/f079c46c-3b61-4541-9979-1ebb3f8b07ad/556740c9-c7d4-4bae-9265-7611d48d31ef.jpg at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)

And in the server container:

[Nest] 1 - 05/06/2023, 8:42:46 AM LOG [CommunicationGateway] New websocket connection: i8vIc4TCabbBg41kAAED [Nest] 1 - 05/06/2023, 8:43:00 AM LOG [CommunicationGateway] Client i8vIc4TCabbBg41kAAED disconnected from Websocket [Nest] 1 - 05/06/2023, 8:44:01 AM LOG [CommunicationGateway] New websocket connection: XidP-lQ4Qb2UIrNsAAEF [Nest] 1 - 05/06/2023, 8:44:03 AM LOG [CommunicationGateway] Client XidP-lQ4Qb2UIrNsAAEF disconnected from Websocket [Nest] 1 - 05/06/2023, 9:02:45 AM LOG [CommunicationGateway] New websocket connection: mwFusSRRi-JiMogIAAEH [Nest] 1 - 05/06/2023, 9:02:52 AM LOG [CommunicationGateway] Client mwFusSRRi-JiMogIAAEH disconnected from Websocket [Nest] 1 - 05/06/2023, 9:03:31 AM LOG [CommunicationGateway] New websocket connection: QeQVRLmtV9slitntAAEJ [Nest] 1 - 05/06/2023, 9:03:31 AM LOG [CommunicationGateway] New websocket connection: TJh5eZBk1EVji81XAAEL [Nest] 1 - 05/06/2023, 9:03:32 AM LOG [CommunicationGateway] New websocket connection: 68wNf0GPzJQmRWvdAAEN [Nest] 1 - 05/06/2023, 9:03:41 AM LOG [CommunicationGateway] Client 68wNf0GPzJQmRWvdAAEN disconnected from Websocket [Nest] 1 - 05/06/2023, 9:05:40 AM LOG [CommunicationGateway] Client QeQVRLmtV9slitntAAEJ disconnected from Websocket [Nest] 1 - 05/06/2023, 9:05:40 AM LOG [CommunicationGateway] Client TJh5eZBk1EVji81XAAEL disconnected from Websocket [Nest] 1 - 05/06/2023, 9:09:27 AM LOG [CommunicationGateway] New websocket connection: 1_93BQ2SjzGRR6kgAAER [Nest] 1 - 05/06/2023, 9:09:27 AM LOG [CommunicationGateway] New websocket connection: ZpZvr4Knr9M7T5fZAAES [Nest] 1 - 05/06/2023, 9:09:27 AM LOG [CommunicationGateway] New websocket connection: nEF9GbFoozmOWe0lAAET [Nest] 1 - 05/06/2023, 9:09:29 AM LOG [CommunicationGateway] Client nEF9GbFoozmOWe0lAAET disconnected from Websocket [Nest] 1 - 05/06/2023, 9:11:59 AM LOG [CommunicationGateway] Client 1_93BQ2SjzGRR6kgAAER disconnected from Websocket [Nest] 1 - 05/06/2023, 9:11:59 AM LOG [CommunicationGateway] Client ZpZvr4Knr9M7T5fZAAES disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:00 AM LOG [CommunicationGateway] New websocket connection: fiQkaTZ_hs5IsS9PAAEW [Nest] 1 - 05/06/2023, 9:49:00 AM LOG [CommunicationGateway] New websocket connection: i-69PnXbg1oL-B2QAAEX [Nest] 1 - 05/06/2023, 9:49:09 AM LOG [CommunicationGateway] Client fiQkaTZ_hs5IsS9PAAEW disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:09 AM LOG [CommunicationGateway] Client i-69PnXbg1oL-B2QAAEX disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:41 AM LOG [CommunicationGateway] New websocket connection: V5Pu_jDUxGV1x8EjAAEZ [Nest] 1 - 05/06/2023, 9:49:42 AM LOG [CommunicationGateway] New websocket connection: dnylbhToTz-3LJtqAAEb [Nest] 1 - 05/06/2023, 9:49:42 AM LOG [CommunicationGateway] New websocket connection: yMy_ElpP_hzAIYMBAAEd [Nest] 1 - 05/06/2023, 9:49:51 AM LOG [CommunicationGateway] Client yMy_ElpP_hzAIYMBAAEd disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:52 AM LOG [CommunicationGateway] Client dnylbhToTz-3LJtqAAEb disconnected from Websocket [Nest] 1 - 05/06/2023, 9:49:52 AM LOG [CommunicationGateway] Client V5Pu_jDUxGV1x8EjAAEZ disconnected from Websocket [Nest] 1 - 05/08/2023, 5:21:32 AM LOG [CommunicationGateway] New websocket connection: LwpGouSHwL5HL-yIAAEf [Nest] 1 - 05/08/2023, 5:21:37 AM LOG [CommunicationGateway] Client LwpGouSHwL5HL-yIAAEf disconnected from Websocket [Nest] 1 - 05/08/2023, 5:21:52 AM LOG [CommunicationGateway] New websocket connection: 5ujGEioyrElFQC40AAEh [Nest] 1 - 05/08/2023, 5:21:52 AM LOG [CommunicationGateway] Client 5ujGEioyrElFQC40AAEh disconnected from Websocket [Nest] 1 - 05/08/2023, 5:21:55 AM LOG [CommunicationGateway] New websocket connection: nEewhxlm7vpVWUlgAAEj [Nest] 1 - 05/08/2023, 5:22:08 AM LOG [CommunicationGateway] Client nEewhxlm7vpVWUlgAAEj disconnected from Websocket [Nest] 1 - 05/08/2023, 5:24:18 AM LOG [CommunicationGateway] New websocket connection: CiWiEIrSDxdiZhWFAAEl [Nest] 1 - 05/08/2023, 5:24:25 AM LOG [CommunicationGateway] Client CiWiEIrSDxdiZhWFAAEl disconnected from Websocket [Nest] 1 - 05/08/2023, 5:26:19 AM LOG [CommunicationGateway] New websocket connection: WgFqHf80Va-AHnw-AAEn [Nest] 1 - 05/08/2023, 5:26:19 AM LOG [CommunicationGateway] Client WgFqHf80Va-AHnw-AAEn disconnected from Websocket

No real change either. The generate thumbnail job is stalled at 1223 waiting though...

wouterverduin commented 1 year ago

@alextran1502

Update: after some time the generate thumbnail task finished and somehow all the photos where showing on the web ans had the checkmark in the app! Later i made some more photos and noticed it, again, missed some photos. Redid the thumbnail task which finished fast this time and it was all sorted again.

Remains the question: how and why?

For now the workaround is acceptable but is there a way to automate that task?

alextran1502 commented 1 year ago

@wouterverduin can you check the microservices container log to find out why it wasn't generating the thumbnail for those photos?

esmoyer commented 1 year ago

Manually running the missing thumbnails job fixed this for me as well

alextran1502 commented 1 year ago

Should have a better indication of uploaded files without thumbnail, which is implemented in #2561