search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
53.46k stars 2.83k forks source link

Many photos and videos showing "Error Loading Image" #12472

Closed elmuziko closed 2 months ago

elmuziko commented 2 months ago

The bug

Several of my photos now show Error Loading Image as a thumbnail from the desktop version. When I click the photo/video, it loads without issue. On my (iOS) mobile app I have an exclamation mark. If the item is a video it will play without issue, if a photo I get a file type image with a line through it.

Desktop: image

Mobile: 20240908_133953000_iOS

You'll have to forgive me if the docker-compose.yml content isn't exactly what you're after. I'm on unraid which doesn't use it out of the box. I got this using docker inspect <ID>

The OS that Immich Server is running on

Unraid 6.12.10

Version of Immich Server

v1.114.0

Version of Immich Mobile App

1.113.1 build 172

Platform with the issue

Your docker-compose.yml content

[
    {
        "Id": "ae47b54025a37155cb76141c29c15ccaad2b5e71409b583098dfbbdced07a749",
        "Created": "2024-09-07T16:11:31.852064468Z",
        "Path": "/init",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 28715,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-09-08T03:48:51.684742989Z",
            "FinishedAt": "2024-09-08T03:48:21.51314524Z"
        },
        "Image": "sha256:d4a9962003b2831f84171e215c3c499b79fba1cb26bfcb764beb0f2c0486a9cf",
        "ResolvConfPath": "/var/lib/docker/containers/ae47b54025a37155cb76141c29c15ccaad2b5e71409b583098dfbbdced07a749/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/ae47b54025a37155cb76141c29c15ccaad2b5e71409b583098dfbbdced07a749/hostname",
        "HostsPath": "/var/lib/docker/containers/ae47b54025a37155cb76141c29c15ccaad2b5e71409b583098dfbbdced07a749/hosts",
        "LogPath": "/var/lib/docker/containers/ae47b54025a37155cb76141c29c15ccaad2b5e71409b583098dfbbdced07a749/ae47b54025a37155cb76141c29c15ccaad2b5e71409b583098dfbbdced07a749-json.log",
        "Name": "/immich",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/mnt/user/Photos/Immich/:/photos:rw",
                "/mnt/user/Photos/Photos/:/import:ro",
                "/mnt/user/appdata/immich:/config:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "1",
                    "max-size": "50m"
                }
            },
            "NetworkMode": "bridge",
            "PortBindings": {
                "8080/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8086"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mnt/user/Photos/Photos",
                "Destination": "/import",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/mnt/user/appdata/immich",
                "Destination": "/config",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "f38f13086ea7faad64cffa99f3f04af936b3db0442e1fd8510f7dbf71947a830",
                "Source": "/var/lib/docker/volumes/f38f13086ea7faad64cffa99f3f04af936b3db0442e1fd8510f7dbf71947a830/_data",
                "Destination": "/libraries",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/mnt/user/Photos/Immich",
                "Destination": "/photos",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "ae47b54025a3",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8080/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "MACHINE_LEARNING_WORKERS=1",
                "MACHINE_LEARNING_WORKER_TIMEOUT=120",
                "PUID=99",
                "PGID=100",
                "UMASK=022",
                "REDIS_PASSWORD=REMOVED",
                "HOST_HOSTNAME=Tower",
                "HOST_CONTAINERNAME=immich",
                "DB_HOSTNAME=192.168.4.26",
                "DB_USERNAME=REMOVED",
                "DB_PORT=5433",
                "REDIS_PORT=6379",
                "MACHINE_LEARNING_GPU_ACCELERATION=",
                "TZ=Europe/London",
                "DB_DATABASE_NAME=REMOVED",
                "DB_PASSWORD=REMOVED",
                "REDIS_HOSTNAME=192.168.4.26",
                "HOST_OS=Unraid",
                "PATH=/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HOME=/root",
                "LANGUAGE=en_US.UTF-8",
                "LANG=en_US.UTF-8",
                "TERM=xterm",
                "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0",
                "S6_VERBOSITY=1",
                "S6_STAGE2_HOOK=/docker-mods",
                "VIRTUAL_ENV=/lsiopy",
                "IMMICH_BUILD_DATA=/app/immich/server",
                "IMMICH_ENV=production",
                "IMMICH_MACHINE_LEARNING_URL=http://127.0.0.1:3003",
                "IMMICH_MEDIA_LOCATION=/photos",
                "IMMICH_PORT=8080",
                "MACHINE_LEARNING_CACHE_FOLDER=/config/machine-learning/models",
                "NVIDIA_DRIVER_CAPABILITIES=compute,video,utility",
                "TRANSFORMERS_CACHE=/config/machine-learning/models",
                "NODE_ENV=production"
            ],
            "Cmd": null,
            "Image": "ghcr.io/imagegenius/immich",
            "Volumes": {
                "/config": {},
                "/libraries": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "/init"
            ],
            "OnBuild": null,
            "Labels": {
                "build_version": "ImageGenius Version:- v1.114.0-ig323 Build-date:- 2024-09-07T01:00:07+10:00",
                "maintainer": "hydazz, martabal",
                "net.unraid.docker.icon": "https://raw.githubusercontent.com/imagegenius/templates/main/unraid/img/immich.png",
                "net.unraid.docker.managed": "dockerman",
                "net.unraid.docker.webui": "http://[IP]:[PORT:8080]",
                "org.opencontainers.image.authors": "imagegenius.io",
                "org.opencontainers.image.created": "2024-09-07T01:00:07+10:00",
                "org.opencontainers.image.description": "Immich is a high performance self-hosted photo and video backup solution.",
                "org.opencontainers.image.documentation": "https://docs.linuxserver.io/images/docker-baseimage-ubuntu",
                "org.opencontainers.image.licenses": "GPL-3.0-only",
                "org.opencontainers.image.ref.name": "ebfadeedba5497e5ddb49bcfb3d0ab1548fec4f6",
                "org.opencontainers.image.revision": "ebfadeedba5497e5ddb49bcfb3d0ab1548fec4f6",
                "org.opencontainers.image.source": "https://github.com/imagegenius/docker-immich",
                "org.opencontainers.image.title": "Immich",
                "org.opencontainers.image.url": "https://github.com/imagegenius/docker-immich/packages",
                "org.opencontainers.image.vendor": "imagegenius.io",
                "org.opencontainers.image.version": "v1.114.0-ig323"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "ee8b9d4eea8f1a52ab1310462b40865d7b047ab2af35c83e95506efaddd13f33",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8086"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8086"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/ee8b9d4eea8f",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "25ee0506fa225da22c745fd1399b060d68e63eeeba737aa7e335d2450451188d",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "468009c5af8e88f33ae71a4f4159d497bc05684ab902230deb83f4e4bccd65d3",
                    "EndpointID": "25ee0506fa225da22c745fd1399b060d68e63eeeba737aa7e335d2450451188d",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

Your .env content

"Env": [
                "MACHINE_LEARNING_WORKERS=1",
                "MACHINE_LEARNING_WORKER_TIMEOUT=120",
                "PUID=99",
                "PGID=100",
                "UMASK=022",
                "REDIS_PASSWORD=REMOVED",
                "HOST_HOSTNAME=Tower",
                "HOST_CONTAINERNAME=immich",
                "DB_HOSTNAME=192.168.4.26",
                "DB_USERNAME=REMOVED",
                "DB_PORT=5433",
                "REDIS_PORT=6379",
                "MACHINE_LEARNING_GPU_ACCELERATION=",
                "TZ=Europe/London",
                "DB_DATABASE_NAME=REMOVED",
                "DB_PASSWORD=REMOVED",
                "REDIS_HOSTNAME=192.168.4.26",
                "HOST_OS=Unraid",
                "PATH=/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HOME=/root",
                "LANGUAGE=en_US.UTF-8",
                "LANG=en_US.UTF-8",
                "TERM=xterm",
                "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0",
                "S6_VERBOSITY=1",
                "S6_STAGE2_HOOK=/docker-mods",
                "VIRTUAL_ENV=/lsiopy",
                "IMMICH_BUILD_DATA=/app/immich/server",
                "IMMICH_ENV=production",
                "IMMICH_MACHINE_LEARNING_URL=http://127.0.0.1:3003",
                "IMMICH_MEDIA_LOCATION=/photos",
                "IMMICH_PORT=8080",
                "MACHINE_LEARNING_CACHE_FOLDER=/config/machine-learning/models",
                "NVIDIA_DRIVER_CAPABILITIES=compute,video,utility",
                "TRANSFORMERS_CACHE=/config/machine-learning/models",
                "NODE_ENV=production"
            ],

Reproduction steps

  1. Updated to v1.114.0
  2. Added new photos, most of which would not load so removed, re-added, and same issue.
  3. Ran Thumbnail generation job, countless older photos suddenly dropped off. ...

Relevant log output

No response

Additional information

Following a few other discussions surrounding this matter, I have tried switching from WEBP to JPG, which did nothing, then converted back. I pushed my docker image back to v1.112.1 but that did far more damage than good so restored. I have a second unraid server I use to back everything up. This is vanilla in that its only use is to be a carbon copy of my main server. This too has inherited this issue meaning messing around with thumbnails and downgrading docker have not caused the problem.

bo0tzz commented 2 months ago

What do the logs say when you run thumbnail generation?

cc @martabal since this could potentially be an AIO issue?

martabal commented 2 months ago

Maybe. But it's hard to tell without the docker logs

cole21771 commented 2 months ago

Several people in this thread (as well as me setting up a brand new instance) are experiencing it too https://github.com/immich-app/immich/discussions/10570

elmuziko commented 2 months ago

[Nest] 211 - 09/08/2024, 5:15:30 PM LOG [Api:EventRepository] Websocket Connect: ZFpVMzxeCb-HO2tlAAAl [Nest] 211 - 09/08/2024, 5:18:45 PM LOG [Api:EventRepository] Websocket Disconnect: ZFpVMzxeCb-HO2tlAAAl [Nest] 211 - 09/08/2024, 5:22:51 PM LOG [Api:EventRepository] Websocket Connect: CoOdZonpq4oR5cA2AAAn [Nest] 211 - 09/08/2024, 5:22:59 PM LOG [Api:EventRepository] Websocket Connect: KEQ0X73CNKH9kOzGAAAp [Nest] 231 - 09/08/2024, 5:23:35 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset bdbf258a-7e1c-4dcc-ba52-aeb61335039d [Nest] 231 - 09/08/2024, 5:23:35 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 146eee1c-e558-47ab-ae44-8c434f64f99d [Nest] 231 - 09/08/2024, 5:23:35 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset e18849a0-6a96-4223-bd1f-a303524b6cff

This is from hitting missing

image

And then it fiiled up with entries like this with zero errors for a good 90 minutes when I hit full.

[Nest] 231 - 09/08/2024, 6:42:15 PM LOG [Microservices:MediaService] Successfully generated WEBP video thumbnail for asset 41a66940-5b16-42ec-b800-76aee5056f58 [Nest] 231 - 09/08/2024, 6:42:15 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 9719dc83-d071-49e0-aff7-63633f954df9 [Nest] 231 - 09/08/2024, 6:42:16 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 677205dd-0df9-4d76-8db5-89d1fa7cb09e [Nest] 231 - 09/08/2024, 6:42:16 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset fe1c3b12-df4a-44f3-99dd-14989b1d6d08 [Nest] 231 - 09/08/2024, 6:42:16 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset b45ef543-a51a-41d2-949b-121f48daf456 [Nest] 231 - 09/08/2024, 6:42:17 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 71e020d9-4997-4c26-b471-1e8b001bb999 [Nest] 231 - 09/08/2024, 6:42:17 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset fabb2e27-ec67-4e65-87ff-ae11c26c6fe7 [Nest] 231 - 09/08/2024, 6:42:17 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 772121aa-75f7-42e2-b981-6ba77a4e5f49 [Nest] 231 - 09/08/2024, 6:42:18 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset ecc3531b-1abd-44c1-adbd-02c28b3ed975 [Nest] 231 - 09/08/2024, 6:42:18 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset a4583ee8-3ef7-4ab0-a125-2da244923576 [Nest] 231 - 09/08/2024, 6:42:18 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset fa9a0d90-a930-4472-8b34-6116419c3551 [Nest] 231 - 09/08/2024, 6:42:20 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 404fa822-a769-4e29-9763-43e66448600a [Nest] 231 - 09/08/2024, 6:42:20 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 53b06de4-c9a8-46e6-8506-b5cc369804f7 [Nest] 231 - 09/08/2024, 6:42:20 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset dedf97db-cf40-48ce-b3ee-14e514309ed5 [Nest] 231 - 09/08/2024, 6:42:20 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset af923464-7e5a-46be-8b23-280105dd891d [Nest] 231 - 09/08/2024, 6:42:21 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 03c1c8b0-80cc-4434-9995-70675d316609 [Nest] 231 - 09/08/2024, 6:42:22 PM LOG [Microservices:MediaService] Successfully generated WEBP video thumbnail for asset 96d039a7-76ea-4ec7-98d9-d51a1698c50a

martabal commented 2 months ago

Several people in this thread (as well as me setting up a brand new instance) are experiencing it too

https://github.com/immich-app/immich/discussions/10570

I don't think it's the same issue and there's no docker logs, I can't really help here

This is from hitting missing

What about waiting for all jobs to be finished?

elmuziko commented 2 months ago

Anything specific you'd like me to run? I have zero jobs waiting at the moment so can run whatever you like.

I just got this whilst navigating Immich.

[Nest] 211 - 09/08/2024, 7:19:37 PM LOG [Api:EventRepository] Websocket Connect: -bZRDEhs5ohRoq-EAAAz [Nest] 211 - 09/08/2024, 7:19:50 PM ERROR [Api:LoggerRepository~sqgtv454] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/2f/bf/2fbf20c3-d032-40e7-bf97-23b735b43e98-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:19:50 PM ERROR [Api:GlobalExceptionFilter~sqgtv454] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/2f/bf/2fbf20c3-d032-40e7-bf97-23b735b43e98-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:19:52 PM ERROR [Api:LoggerRepository~3htp7cs5] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/d6/19/d6196f45-1286-44c3-8498-cbdaf30d1b4d-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:19:52 PM ERROR [Api:GlobalExceptionFilter~3htp7cs5] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/d6/19/d6196f45-1286-44c3-8498-cbdaf30d1b4d-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:19:59 PM ERROR [Api:LoggerRepository~e965q4cv] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:19:59 PM ERROR [Api:GlobalExceptionFilter~e965q4cv] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:03 PM ERROR [Api:LoggerRepository~gkh8wlur] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/23/4f/234f41fd-5dbe-4379-b844-f79b0d570473-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:03 PM ERROR [Api:GlobalExceptionFilter~gkh8wlur] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/23/4f/234f41fd-5dbe-4379-b844-f79b0d570473-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:05 PM ERROR [Api:LoggerRepository~zfa9j1bi] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/c2/b3/c2b32776-3b3c-4b6a-a53a-6e7bc335ab0f-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:05 PM ERROR [Api:GlobalExceptionFilter~zfa9j1bi] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/c2/b3/c2b32776-3b3c-4b6a-a53a-6e7bc335ab0f-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:05 PM ERROR [Api:LoggerRepository~m4jl3c9e] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/cd/86/cd863f8c-1177-4c36-8861-258bf1ebfba8-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:05 PM ERROR [Api:GlobalExceptionFilter~m4jl3c9e] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/cd/86/cd863f8c-1177-4c36-8861-258bf1ebfba8-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:08 PM ERROR [Api:LoggerRepository~a9et9bkm] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/c2/b3/c2b32776-3b3c-4b6a-a53a-6e7bc335ab0f-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:08 PM ERROR [Api:GlobalExceptionFilter~a9et9bkm] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/c2/b3/c2b32776-3b3c-4b6a-a53a-6e7bc335ab0f-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:08 PM ERROR [Api:LoggerRepository~i8sp939a] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/cd/86/cd863f8c-1177-4c36-8861-258bf1ebfba8-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:08 PM ERROR [Api:GlobalExceptionFilter~i8sp939a] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/cd/86/cd863f8c-1177-4c36-8861-258bf1ebfba8-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:21 PM ERROR [Api:LoggerRepository~yqo3i778] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:21 PM ERROR [Api:GlobalExceptionFilter~yqo3i778] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:57 PM ERROR [Api:LoggerRepository~rd8ix0gg] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/2f/bf/2fbf20c3-d032-40e7-bf97-23b735b43e98-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:57 PM ERROR [Api:GlobalExceptionFilter~rd8ix0gg] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/2f/bf/2fbf20c3-d032-40e7-bf97-23b735b43e98-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:20:58 PM ERROR [Api:LoggerRepository~54ji0h8t] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/d6/19/d6196f45-1286-44c3-8498-cbdaf30d1b4d-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:20:58 PM ERROR [Api:GlobalExceptionFilter~54ji0h8t] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/d6/19/d6196f45-1286-44c3-8498-cbdaf30d1b4d-thumbnail.webp'

[Nest] 211 - 09/08/2024, 7:21:02 PM ERROR [Api:LoggerRepository~3harszlb] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9)

[Nest] 211 - 09/08/2024, 7:21:02 PM ERROR [Api:GlobalExceptionFilter~3harszlb] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp'`

martabal commented 2 months ago

You have missing thumbnails, you have no assets generated when clicking on "missing" for the "generating thumbnails" job ? If no, you probably want to re-generate all thumbnails

elmuziko commented 2 months ago

So I've just hit Missing thumbnail creation three times with small stints between each click.

[Nest] 231 - 09/08/2024, 7:50:13 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 2b35dcfa-681c-4d99-a9e8-835b168e7cf8 [Nest] 231 - 09/08/2024, 7:50:13 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset b2a336fb-fee5-46d9-8f21-fcb6e92bfb00 [Nest] 231 - 09/08/2024, 7:50:13 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset f40a0d12-bb4a-4b27-a522-19fec86f4eb7 [Nest] 231 - 09/08/2024, 7:50:14 PM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 2e319d4e-920a-4824-a649-dcb055f1a4f9 [Nest] 231 - 09/08/2024, 7:50:15 PM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 65b14cd9-c981-4dee-b1b6-9fcce65436fb [Nest] 231 - 09/08/2024, 7:50:15 PM LOG [Microservices:MediaService] Successfully generated JPEG video preview for asset c9bf414f-4fc9-4626-8d13-76070b3b2787 [Nest] 231 - 09/08/2024, 7:50:15 PM LOG [Microservices:MediaService] Successfully generated JPEG video preview for asset c7aae019-2a9d-418f-9a12-35f926e233f6 [Nest] 231 - 09/08/2024, 7:50:17 PM LOG [Microservices:MediaService] Successfully generated WEBP video thumbnail for asset c7aae019-2a9d-418f-9a12-35f926e233f6 [Nest] 231 - 09/08/2024, 7:51:59 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset bc581bc1-42ca-4d6d-bd42-40500562fcd4 [Nest] 231 - 09/08/2024, 7:51:59 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 4cc7c83b-df59-4933-98a5-044eda6595c7 [Nest] 231 - 09/08/2024, 7:51:59 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 4d43f45a-f4b4-417e-b9db-a9eb0a99677b [Nest] 231 - 09/08/2024, 7:52:00 PM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 229800a7-b1b0-41be-9529-a8ae92aa2d40 [Nest] 231 - 09/08/2024, 7:52:00 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 56d63af1-02b3-4419-aa25-a70e55589808 [Nest] 231 - 09/08/2024, 7:52:00 PM LOG [Microservices:MediaService] Successfully generated WEBP video thumbnail for asset b5facfcd-7653-44dc-a9d0-4d8538b98623 [Nest] 231 - 09/08/2024, 7:52:01 PM LOG [Microservices:MediaService] Successfully generated JPEG video preview for asset cc4fdf46-cb56-411b-a1d0-1fac296e7767 [Nest] 231 - 09/08/2024, 7:52:01 PM LOG [Microservices:MediaService] Successfully generated JPEG video preview for asset 71b0bbd0-ea1b-4942-871c-ffa0a7f64617 [Nest] 231 - 09/08/2024, 7:52:03 PM LOG [Microservices:MediaService] Successfully generated WEBP video thumbnail for asset 71b0bbd0-ea1b-4942-871c-ffa0a7f64617 [Nest] 231 - 09/08/2024, 7:54:14 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 0ceb5b43-4e4e-4ce6-b148-5a0cb83d977e [Nest] 231 - 09/08/2024, 7:54:15 PM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 0826d758-6668-4ec7-9292-b10ee8247970 [Nest] 231 - 09/08/2024, 7:54:15 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset bc1ceced-ce8e-4417-b997-8ebbedf29b5c [Nest] 231 - 09/08/2024, 7:54:15 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 0f0d05c6-2123-4fbb-add2-170607acda96 [Nest] 231 - 09/08/2024, 7:56:11 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 7041f250-03a1-4001-b4e3-3836747b69d3 [Nest] 231 - 09/08/2024, 7:56:12 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 2244a60a-8622-4470-9b7e-65bb7519791c [Nest] 231 - 09/08/2024, 7:56:12 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 5f3af3b5-67b6-4b9c-b430-c2a92ad58e3d

No errors. Assuming it's spotting ones without thumbnails on each run?

I have a backup of everything so am willing to try unconventional things.

aelshaabany-wolf commented 2 months ago

I have the same issue like broken thumbnails. And this happened as follows:

So what happens after upgrading versions? Is DB references got broken or what ?

etnoy commented 2 months ago

Just to double-check, does anything change if you make a full restart of Immich, i.e. docker compose down, wait, then docker compose up. I had a strange issue on my first boot of v.114 but the above restart fixed it (strangely enough)

hkrob commented 2 months ago

Same, just updated. image

Environment is running via the official docker-compose instructions on unraid

Tried docker compose down/ up - no luck. No pending jobs If i download the original image - it works. Would suggest making logs available from the GUI...

Going to re-gen all thumbs now and see what happens Edit: Hitting regenerate all thumbs did nothing - the job didn't seem to start

This shows in the logs when I try to generate thumbs

`[Nest] 7 - 09/09/2024, 2:45:16 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/queue-generate-thumbnails): QueryFailedError: column asset.previewPath does not exist [Nest] 7 - 09/09/2024, 2:45:16 AM ERROR [Microservices:JobService] QueryFailedError: column asset.previewPath does not exist at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async SelectQueryBuilder.loadRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2192:25) at async SelectQueryBuilder.executeEntitiesAndRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2040:26) at async SelectQueryBuilder.getRawAndEntities (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:684:29) at async SelectQueryBuilder.getMany (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:750:25) at async paginatedBuilder (/usr/src/app/dist/utils/pagination.js:43:19) at async usePagination (/usr/src/app/dist/utils/pagination.js:16:24) at async MediaService.handleQueueGenerateThumbnails (/usr/src/app/dist/services/media.service.js:64:26) at async /usr/src/app/dist/services/job.service.js:148:36 [Nest] 7 - 09/09/2024, 2:45:16 AM ERROR [Microservices:JobService] Object: { "force": true }

[Nest] 7 - 09/09/2024, 2:47:04 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/queue-generate-thumbnails): QueryFailedError: column AssetEntity.previewPath does not exist [Nest] 7 - 09/09/2024, 2:47:04 AM ERROR [Microservices:JobService] QueryFailedError: column AssetEntity.previewPath does not exist at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async SelectQueryBuilder.loadRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2192:25) at async SelectQueryBuilder.executeEntitiesAndRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2040:26) at async SelectQueryBuilder.getRawAndEntities (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:684:29) at async SelectQueryBuilder.getMany (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:750:25) at async paginate (/usr/src/app/dist/utils/pagination.js:28:19) at async usePagination (/usr/src/app/dist/utils/pagination.js:16:24) at async MediaService.handleQueueGenerateThumbnails (/usr/src/app/dist/services/media.service.js:64:26) at async /usr/src/app/dist/services/job.service.js:148:36 [Nest] 7 - 09/09/2024, 2:47:04 AM ERROR [Microservices:JobService] Object: { "force": false }`

elmuziko commented 2 months ago

Just to double-check, does anything change if you make a full restart of Immich, i.e. docker compose down, wait, then docker compose up. I had a strange issue on my first boot of v.114 but the above restart fixed it (strangely enough)

Sadly, not for me.

Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] BullModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] OpenTelemetryModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] TypeOrmModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] OpenTelemetryCoreModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] ConfigHostModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] DiscoveryModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] ClsModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] ConfigModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] EventEmitterModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] BullModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] BullModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] TypeOrmCoreModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] TypeOrmModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:InstanceLoader] MicroservicesModule dependencies initialized [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:NestApplication] Nest application successfully started [Nest] 255 - 09/09/2024, 8:56:56 AM LOG [Microservices:Bootstrap] Immich Microservices is running [v1.114.0] [PRODUCTION] [09/09/24 08:57:00] INFO Started server process [297] [09/09/24 08:57:00] INFO Waiting for application startup. [09/09/24 08:57:00] INFO Created in-memory cache with unloading after 300s of inactivity. [09/09/24 08:57:00] INFO Initialized request thread pool with 4 threads. [09/09/24 08:57:00] INFO Application startup complete. [Nest] 223 - 09/09/2024, 8:58:08 AM LOG [Api:EventRepository] Websocket Connect: NkLthy_Zeljx7hnZAAAD [Nest] 255 - 09/09/2024, 8:58:13 AM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 467091aa-e281-4445-b81a-5b5130301d17 [Nest] 255 - 09/09/2024, 8:58:14 AM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 49c26a0f-e95e-417e-8eb3-44ff92c86cf2 [Nest] 255 - 09/09/2024, 8:58:15 AM LOG [Microservices:MediaService] Successfully generated WEBP video thumbnail for asset c9bf414f-4fc9-4626-8d13-76070b3b2787 [Nest] 255 - 09/09/2024, 8:58:15 AM LOG [Microservices:MediaService] Successfully generated JPEG video preview for asset 94993d8c-c843-4d96-96f1-1ebd007759fd [Nest] 255 - 09/09/2024, 8:58:15 AM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 9cf55af9-96ec-4290-8673-61dc98dff216 [Nest] 223 - 09/09/2024, 8:58:56 AM LOG [Api:EventRepository] Websocket Disconnect: WRx8_IJVfRZgTWzNAAAB [Nest] 223 - 09/09/2024, 8:58:57 AM ERROR [Api:LoggerRepository~2bxl4iwv] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9) [Nest] 223 - 09/09/2024, 8:58:57 AM ERROR [Api:GlobalExceptionFilter~2bxl4iwv] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/01/be/01bed5af-a3da-4846-9fa0-43c880783a35-thumbnail.webp' [Nest] 223 - 09/09/2024, 8:58:57 AM LOG [Api:EventRepository] Websocket Connect: ilnuKr8QH_yqtzidAAAF

The errors are generated when I refresh the webpage.

aelshaabany-wolf commented 2 months ago

Just to double-check, does anything change if you make a full restart of Immich, i.e. docker compose down, wait, then docker compose up. I had a strange issue on my first boot of v.114 but the above restart fixed it (strangely enough)

I can't confirm this as I just did a re-generate all, as I faced the same issue when upgraded to v1.113 and ran re-generate and it fixed the issue for me.

hkrob commented 2 months ago

My problem is I cannot re-create thumbnails, the following error occurs

[Nest] 7  - 09/10/2024, 1:21:41 AM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/queue-generate-thumbnails): QueryFailedError: column asset.previewPath does not exist
[Nest] 7  - 09/10/2024, 1:21:41 AM   ERROR [Microservices:JobService] QueryFailedError: column asset.previewPath does not exist
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async SelectQueryBuilder.loadRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2192:25)
    at async SelectQueryBuilder.executeEntitiesAndRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2040:26)
    at async SelectQueryBuilder.getRawAndEntities (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:684:29)
    at async SelectQueryBuilder.getMany (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuilder.js:750:25)
    at async paginatedBuilder (/usr/src/app/dist/utils/pagination.js:43:19)
    at async usePagination (/usr/src/app/dist/utils/pagination.js:16:24)
    at async MediaService.handleQueueGenerateThumbnails (/usr/src/app/dist/services/media.service.js:64:26)
    at async /usr/src/app/dist/services/job.service.js:148:36
[Nest] 7  - 09/10/2024, 1:21:41 AM   ERROR [Microservices:JobService] Object:
{
  "force": true
}
BlackRockCity commented 2 months ago

This may or may not be related to the original post. I'm brand new to Immich. Using v1.114.0 cuda.

I connected my immich docker to a few NFS mounts to store photos, libraries, and config. I noted that when I uploaded from the iPhone while browsing the library on the immich web GUI it would cause thumbnail 'error loading image' errors. About half of 75 images had the error.

I suspected that if corruption is taking place, its from some sort of contention issue/atomic writes/database locks I don't know. Something about simultaneously hitting it from the phone and desktop was causing errors.

I noticed that my NFS options were set to nolock, soft. So I removed those which defaults NFS to lock, hard. Then I performed the exact same upload test and all thumbnails loaded correctly except one image out of 75. A dramatic improvement over the first test.

I noted that performance went way down when I removed those NFS options. Now I'm tempted to rebuild the system with iSCSI mounts instead of using NFS. Traditionally, for my docker needs, NFS has been fast enough and very reliable. iSCSI has been buggy and difficult to use in the past.

  1. nolock Option: • Description: The nolock option disables file locking for NFS mounts. NFS normally uses a locking protocol to coordinate access to files among multiple clients. When nolock is enabled, this locking mechanism is turned off. • Impact: Disabling locking can lead to potential data consistency issues when multiple clients attempt to access or modify the same file concurrently. However, in some environments, particularly with older or incompatible NFS clients and servers, file locking may not work properly, leading to the use of the nolock option to bypass these issues.

  2. soft Option: • Description: The soft option allows the NFS client to handle server access timeouts more gracefully. When an NFS client is configured with the soft option, if an operation times out (e.g., due to network issues or the server being down), the client will receive an error and will not retry the operation indefinitely. • Impact: The soft option can lead to partial or corrupted data being read or written if there is a temporary network interruption, as it will not retry the operation. This could explain issues with partial uploads or errors if the connection to the NFS server is not perfectly stable.

elmuziko commented 2 months ago

I've been doing some more 'stuff' to see if I can better diagnose this.

[Nest] 214 - 09/10/2024, 9:27:03 PM ERROR [Api:LoggerRepository~kx44gtz6] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/12/1a/121a81b3-4e22-40c3-9f15-49250d6a6104-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9) [Nest] 214 - 09/10/2024, 9:27:03 PM ERROR [Api:GlobalExceptionFilter~kx44gtz6] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/12/1a/121a81b3-4e22-40c3-9f15-49250d6a6104-thumbnail.webp' [Nest] 214 - 09/10/2024, 9:27:03 PM ERROR [Api:LoggerRepository~xtp46z3a] Unable to send file: Error Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/1b/e4/1be4cbcb-1649-49d9-91e6-3b14f28afd8f-thumbnail.webp' at async access (node:internal/fs/promises:606:10) at async sendFile (/app/immich/server/dist/utils/file.js:55:9) at async AssetMediaController.viewAsset (/app/immich/server/dist/controllers/asset-media.controller.js:57:9) [Nest] 214 - 09/10/2024, 9:27:03 PM ERROR [Api:GlobalExceptionFilter~xtp46z3a] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/1b/e4/1be4cbcb-1649-49d9-91e6-3b14f28afd8f-thumbnail.webp' [Nest] 236 - 09/10/2024, 9:29:01 PM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: Expected integer for width but received -6 of type number [Nest] 236 - 09/10/2024, 9:29:01 PM ERROR [Microservices:JobService] Error: Expected integer for width but received -6 of type number at Object.invalidParameterError (/app/immich/server/node_modules/sharp/lib/is.js:135:10) at Sharp.<anonymous> (/app/immich/server/node_modules/sharp/lib/resize.js:483:16) at Array.forEach (<anonymous>) at Sharp.extract (/app/immich/server/node_modules/sharp/lib/resize.js:478:38) at MediaRepository.generateThumbnail (/app/immich/server/dist/repositories/media.repository.js:60:22) at PersonService.handleGeneratePersonThumbnail (/app/immich/server/dist/services/person.service.js:453:36) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /app/immich/server/dist/services/job.service.js:148:36 at async Worker.processJob (/app/immich/server/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) at async Worker.retryIfFailed (/app/immich/server/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) [Nest] 236 - 09/10/2024, 9:29:01 PM ERROR [Microservices:JobService] Object: { "id": "2883934a-d613-494f-9eef-ed554325f01e" }

[Nest] 236 - 09/10/2024, 9:27:54 PM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 088cd038-95d4-43f4-967c-c325cf8033c8 [Nest] 236 - 09/10/2024, 9:27:54 PM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 001a226f-4a02-4022-8588-aa812b910b98 [Nest] 236 - 09/10/2024, 9:27:54 PM LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset 1d017ebe-8bc5-4b0a-b47f-095ab65e25cb [Nest] 236 - 09/10/2024, 9:28:15 PM LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset 7f8e1eff-c84b-4359-a9cc-1fd4ec0ba42f

This one here, [Nest] 214 - 09/10/2024, 10:02:41 PM ERROR [Api:GlobalExceptionFilter~zbgapkvo] Unknown error: Error: ENOENT: no such file or directory, access '/photos/thumbs/2dd27ee2-e895-4b8f-9de4-ff49cb9ed7fe/2b/fd/2bfd7263-777c-4c56-9e1e-2b896e254407-thumbnail.webp' I found the folder location in Unraid.

image

For pure testing purposes I copied an existing webp file into this directory and renamed it without issue. The photo then showed a thumbnail albeit the wrong one (which was expected) Point is, it looks like the access is fine, just it doesn;t exist and doesn't want to create it, no matter how many times I hit generate thumbnails.

tbelway commented 2 months ago

I'm experiencing teh same problem but when I generate missing thumbnails it resolves. It seems like whatever hook was working prior on upload is no longer working.

Tailing the container log when I upload a photo from the mobile app results in... no logs. Nothing. It just... doesn't do anything, for both microservices and ml. Naturally, as I'm using the mobile app for this, the webserver container also has no logs.

Interestingly, watching the jobs from admin console the extract metadata job does activate but the generate thumbnails job does not.

I suspect whatever hooks in the thumbnail generation on upload is silently failing.

yydcool commented 2 months ago

Same issue for me on web (client using chrome browser 128). It seems if I scroll the page very fast, then there will be a lot of thumbnails showing ["Error Loading Image"] But if I scroll slowly, all thumbnails image can load without issue.

yydcool commented 2 months ago

oh! this is probably because of cloudflare or nginx that I'm using. without cloudflare: I tried visiting my server directly, the thumbnails can load without issue. But since I'm using a mechanical drive, some thumbnails load with ~1 sec delay. But they can all loaded, even if I scroll fast.

with nginx+cloudflare: if I scroll the page very fast, then there will be a lot of thumbnails showing ["Error Loading Image"] But if I scroll slowly, all thumbnails image can load without issue.