search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
43.12k stars 2.1k forks source link

ffmpeg errors in logs, jobs killing themselves #8410

Closed mich2k closed 4 months ago

mich2k commented 5 months ago

The bug

Hello, first of all I already tried to open a discussion here but due the amount of errors I am getting I really need to understand what is happening

What is happening to me is that, everytime I click "Missing" on library, on transcoding, face det/rec there are always some thousand elements missing for that job, the task always stop and get killed at sometime and feels indeed super unreliable

Today I got some new errors


[Nest] 272  - 03/31/2024, 3:09:23 AM   ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
/photos/encoded-video/4accdf5a-776b-43ee-9377-887c9f2ff237/3a/2a/3a2ad7ef-5697-41dd-baf7-ba829dd6bdb6-MP.mp4: No such file or directory

    at ChildProcess.<anonymous> (/app/immich/server/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 272  - 03/31/2024, 3:09:23 AM   ERROR [JobService] Object:
{
  "id": "3a2ad7ef-5697-41dd-baf7-ba829dd6bdb6"
}

[Nest] 272  - 03/31/2024, 3:09:23 AM   ERROR [JobService] Unable to run job handler (videoConversion/video-conversion): Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
/photos/encoded-video/4accdf5a-776b-43ee-9377-887c9f2ff237/7d/59/7d59af27-023c-4850-845c-c7619dbe54f0-MP.mp4: No such file or directory

[Nest] 272  - 03/31/2024, 3:09:23 AM   ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
/photos/encoded-video/4accdf5a-776b-43ee-9377-887c9f2ff237/7d/59/7d59af27-023c-4850-845c-c7619dbe54f0-MP.mp4: No such file or directory

    at ChildProcess.<anonymous> (/app/immich/server/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 272  - 03/31/2024, 3:09:23 AM   ERROR [JobService] Object:
{
  "id": "7d59af27-023c-4850-845c-c7619dbe54f0"
}

[Nest] 208  - 03/31/2024, 3:13:36 AM     LOG [EventRepository] Websocket Connect:    ixAeG1tu004vnUBqAAAV
[Nest] 208  - 03/31/2024, 3:13:39 AM     LOG [EventRepository] Websocket Disconnect: ixAeG1tu004vnUBqAAAV
[Nest] 208  - 03/31/2024, 3:13:44 AM     LOG [EventRepository] Websocket Connect:    dKZDBQPiuSLHDuqJAAAX
[Nest] 208  - 03/31/2024, 3:13:48 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:06 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:09 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:13 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:15 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:35 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:36 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:37 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:57 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:58 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:58 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:59 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:14:59 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 208  - 03/31/2024, 3:15:23 AM    WARN [AuthGuard] Denied access to admin only route: /api/jobs/videoConversion

All of this from the admin account Does everybody experience these problems? Missing files errors, ffmpeg errors, device creation errors? Thank you all

The OS that Immich Server is running on

Ubuntu Server 22.04.4 LTS

Version of Immich Server

1.100

Version of Immich Mobile App

1.100.0

Platform with the issue

Your docker-compose.yml content

default

Your .env content

default

Reproduction steps

Start any kind of job from the admin panel

Additional information

No response

mertalev commented 5 months ago

If ffprobe is erroring that a file in the encoded-videos folder doesn't exist, then that implies the original asset comes from this folder. That shouldn't happen since this folder is only supposed to store transcodes.

There was a bug a few weeks ago around external libraries importing transcodes and thumbnails as assets, so I wonder if this is what you're seeing. Do you have any external libraries?

mich2k commented 5 months ago

If ffprobe is erroring that a file in the encoded-videos folder doesn't exist, then that implies the original asset comes from this folder. That shouldn't happen since this folder is only supposed to store transcodes.

There was a bug a few weeks ago around external libraries importing transcodes and thumbnails as assets, so I wonder if this is what you're seeing. Do you have any external libraries?

Hello, almost all my assets are stored as external, I was just starting to use the immich upload feature

mertalev commented 5 months ago

Do any of the import paths for the libraries contain the folder /usr/src/app/upload? e.g. ., /usr/src/app, etc. The bug was that import paths like this were importing the generated thumbnails and transcodes as new assets.

mich2k commented 5 months ago

Do any of the import paths for the libraries contain the folder /usr/src/app/upload? e.g. ., /usr/src/app, etc. The bug was that import paths like this were importing the generated thumbnails and transcodes as new assets.

Ok, my setup about paths is a bit convoluted indeed

My volumes are [HOST to CONTAINER]:

/mnt/user/ to /external/root /mnt/user/billy to /external/billy /mnt/user/bob to /external/bob /mnt/user/jack to /external/jack

From the immich admin panel each user has it's /external/name as import paths

Now maybe this is creating a loop somewhere, I did this since I wanted to avoid an user to import in his account another users data (by passing /external/root/ to all users I shall trust each of them to import only their sub path like /external/root/jack but jack is free to import Billy's and Bob's assets)

Now if this is what the issue is and by passing /external/root (and importing /external/root/jack etc for each) to all of them would fix this then is great (even if I don't see how due to my lack of knowledge about the inner workings of immich)

The other mappings are

/mnt/user/immich_encoded_video to /photos/encoded-video: /mnt/user/immich_thumbs to /photos/thumbs/ /mnt/user/immich_libraryto /photos/library

Am I facing this exact bug? Thanks

mich2k commented 4 months ago

Do any of the import paths for the libraries contain the folder /usr/src/app/upload? e.g. ., /usr/src/app, etc. The bug was that import paths like this were importing the generated thumbnails and transcodes as new assets.

Hello, sorry for the poke but might I ask you for the issue ID you were citing here? Thank you

CarterMoody commented 4 months ago

I get the same error here today when uploading. It seems to be an issue when uploading dozens of smaller video files (like live photos). It will chunk through 150 or so uploads in blistering speed, hit 1-2 errors and then bring the entire container to a hault. Only restarting the container in docker helps. For now, uploading large batch 100 files at a time as a workaround

raisinbear commented 4 months ago

Just came here to open an issue about that, but found this. I get this pretty much every time when uploading (iOS) live photos. It doesn't need to be a batch of many assets at all but happens with single uploads reliably. I didn't have a container hault so far, merely find it irritating seeing a bunch of errors regularly. Further I don't see any consequences. The video files belonging to the live photos are encoded successfully and the originals are in their proper place.

mich2k commented 4 months ago

Hello,

my issue was likely due to the missing volume mapping of the upload folder from the immch /library I do still get these errors sometimes but mostly on invalid video files (corrupted, few Bytes of files), I m atm seeking a way to filter them beforehand

If you have specific issues that lead to the same ffmpeg error might be better to open a new issue