search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
51.4k stars 2.72k forks source link

Green diagonal lines with hardware transcoding for portrait videos (rockchip 3588) #10764

Closed gwmullin closed 3 weeks ago

gwmullin commented 4 months ago

The bug

When hardware transcoding is enabled (encoding, the decoding setting doesn't seem to make this better or worse), videos transcoded from an upload in portrait format appear with green/color stripe artifacts, while landscape formats do not. This only occurs with hardware encoding enabled on the rockchip platform. When disabling hardware encoding the issue is no longer present.

With hardware transcoding: Screenshot 2024-07-02 at 9 28 55 AM

Hardware transcoding disabled: Screenshot 2024-07-02 at 10 04 57 AM

The OS that Immich Server is running on

Ubuntu 24.04 LTS (GNU/Linux 6.1.0-1014-rockchip aarch64)

Version of Immich Server

v1.107.0

Version of Immich Mobile App

1.107.0 buil.145

Platform with the issue

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
      service: rkmpp # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - ./immich_config.json:/etc/immich_config.json
      - /nvme/frigate/exports:/exports
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '2.0'

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '2.0'

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:d6c2911ac51b289db208767581a5d154544f2b2fe4914ea5056443f62dc6e900
    healthcheck:
      test: redis-cli ping || exit 1
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '1.0'

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '2.0'

volumes:
  model-cache:

Your .env content

# The location where your uploaded files are stored
UPLOAD_LOCATION=/disks/external_usb
# The location where your database files are stored
DB_DATA_LOCATION=/nvme/immich/postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=America/Los_Angeles

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=<REDACTED>

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. On rockchip platform, enable hardware encoding

2. Upload a video in portrait format (I'm using a pixel 6, resulting in a video uploaded in HVEC in either 1920x1080 or 1440x1080 depending on how it was initiated, both present with issues)

3. Once transcoding is done, there are green-stripe artifacts in the web app view (the mobile app appears to correctly use the original, but I haven't tested in a low-bandwidth remote access scenario)

Note: A video in the same resolution in 'landscape' format does not have this issue. I don't know why this is the case.

Relevant log output

immich_server            | [Nest] 16  - 07/02/2024, 9:58:24 AM   DEBUG [Api:LoggingInterceptor~km2ymohz] POST /api/assets/exist 200 11.76ms 76.21.24.156
immich_server            | [Nest] 16  - 07/02/2024, 9:58:24 AM VERBOSE [Api:LoggingInterceptor~km2ymohz] {"deviceAssetIds":["1000007146"],"deviceId":"544e72a9ef1f32301b08dce5f916787238fe20910b84a60fd5ca0786bae1e0c5"}
immich_server            | [Nest] 16  - 07/02/2024, 9:58:27 AM   DEBUG [Api:LoggingInterceptor~s7w9yfn3] POST /api/assets 201 2546.65ms 76.21.24.156
immich_server            | [Nest] 16  - 07/02/2024, 9:58:27 AM VERBOSE [Api:LoggingInterceptor~s7w9yfn3] {"deviceAssetId":"1000007146","deviceId":"544e72a9ef1f32301b08dce5f916787238fe20910b84a60fd5ca0786bae1e0c5","fileCreatedAt":"2024-07-02T16:58:15.000Z","fileModifiedAt":"2024-07-02T16:58:19.000Z","isFavorite":"false","duration":"0:00:03.000000"}
immich_server            | [Nest] 7  - 07/02/2024, 9:58:27 AM VERBOSE [Microservices:MetadataService] Exif Tags
immich_server            | [Nest] 7  - 07/02/2024, 9:58:27 AM VERBOSE [Microservices:MetadataService] Object:
immich_server            | {
immich_server            |   "SourceFile": "/usr/src/app/upload/upload/f7cecbb4-cf89-45d7-ba78-262c47c574f3/8e/28/8e28affb-2a55-4f49-ae12-66fe26726dfe.mp4",
immich_server            |   "errors": [],
immich_server            |   "tz": "America/Los_Angeles",
immich_server            |   "tzSource": "GPSLatitude/GPSLongitude",
immich_server            |   "Duration": 3.321,
immich_server            |   "PreviewDuration": 0,
immich_server            |   "SelectionDuration": 0,
immich_server            |   "TrackDuration": 3.2992,
immich_server            |   "MediaDuration": 0,
immich_server            |   "GPSLatitude": <REDACTED>,
immich_server            |   "GPSLongitude": <REDACTED>,
immich_server            |   "GPSPosition": "<REDACTED>",
immich_server            |   "ExifToolVersion": 12.85,
immich_server            |   "FileName": "8e28affb-2a55-4f49-ae12-66fe26726dfe.mp4",
immich_server            |   "Directory": "/usr/src/app/upload/upload/f7cecbb4-cf89-45d7-ba78-262c47c574f3/8e/28",
immich_server            |   "FileSize": "5.7 MB",
immich_server            |   "FileModifyDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 19,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 09:58:19-07:00",
immich_server            |     "zoneName": "UTC-7",
immich_server            |     "inferredZone": false
immich_server            |   },
immich_server            |   "FileAccessDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 27,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 09:58:27-07:00",
immich_server            |     "zoneName": "UTC-7",
immich_server            |     "inferredZone": false
immich_server            |   },
immich_server            |   "FileInodeChangeDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 27,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 09:58:27-07:00",
immich_server            |     "zoneName": "UTC-7",
immich_server            |     "inferredZone": false
immich_server            |   },
immich_server            |   "FilePermissions": "-rw-r--r--",
immich_server            |   "FileType": "MP4",
immich_server            |   "FileTypeExtension": "mp4",
immich_server            |   "MIMEType": "video/mp4",
immich_server            |   "MajorBrand": "MP4 Base Media v1 [IS0 14496-12:2003]",
immich_server            |   "MinorVersion": "2.0.0",
immich_server            |   "CompatibleBrands": [
immich_server            |     "isom",
immich_server            |     "iso2",
immich_server            |     "mp41"
immich_server            |   ],
immich_server            |   "MediaDataSize": 5669354,
immich_server            |   "MediaDataOffset": 44,
immich_server            |   "MovieHeaderVersion": 0,
immich_server            |   "CreateDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 15,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 16:58:15",
immich_server            |     "zoneName": "America/Los_Angeles",
immich_server            |     "inferredZone": true
immich_server            |   },
immich_server            |   "ModifyDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 15,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 16:58:15",
immich_server            |     "zoneName": "America/Los_Angeles",
immich_server            |     "inferredZone": true
immich_server            |   },
immich_server            |   "TimeScale": 10000,
immich_server            |   "PreferredRate": 1,
immich_server            |   "PreferredVolume": "100.00%",
immich_server            |   "PreviewTime": "0 s",
immich_server            |   "PosterTime": "0 s",
immich_server            |   "SelectionTime": "0 s",
immich_server            |   "CurrentTime": "0 s",
immich_server            |   "NextTrackID": 6,
immich_server            |   "GPSCoordinates": "<REDACTED>",
immich_server            |   "TrackHeaderVersion": 0,
immich_server            |   "TrackCreateDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 15,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 16:58:15",
immich_server            |     "zoneName": "America/Los_Angeles",
immich_server            |     "inferredZone": true
immich_server            |   },
immich_server            |   "TrackModifyDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 15,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 16:58:15",
immich_server            |     "zoneName": "America/Los_Angeles",
immich_server            |     "inferredZone": true
immich_server            |   },
immich_server            |   "TrackID": 1,
immich_server            |   "TrackLayer": 0,
immich_server            |   "TrackVolume": "0.00%",
immich_server            |   "ImageWidth": 1440,
immich_server            |   "ImageHeight": 1080,
immich_server            |   "GraphicsMode": "srcCopy",
immich_server            |   "OpColor": "0 0 0",
immich_server            |   "CompressorID": "hvc1",
immich_server            |   "SourceImageWidth": 1440,
immich_server            |   "SourceImageHeight": 1080,
immich_server            |   "XResolution": 72,
immich_server            |   "YResolution": 72,
immich_server            |   "BitDepth": 24,
immich_server            |   "PixelAspectRatio": "65536:65536",
immich_server            |   "ColorProfiles": "nclx",
immich_server            |   "ColorPrimaries": "BT.709",
immich_server            |   "TransferCharacteristics": "BT.709",
immich_server            |   "MatrixCoefficients": "BT.709",
immich_server            |   "VideoFullRangeFlag": 1,
immich_server            |   "VideoFrameRate": 29.704,
immich_server            |   "Balance": 0,
immich_server            |   "AudioFormat": "mp4a",
immich_server            |   "AudioChannels": 2,
immich_server            |   "AudioBitsPerSample": 16,
immich_server            |   "AudioSampleRate": 48000,
immich_server            |   "MatrixStructure": "0 1 0 -1 0 0 0 0 1",
immich_server            |   "MediaHeaderVersion": 0,
immich_server            |   "MediaCreateDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 15,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 16:58:15",
immich_server            |     "zoneName": "America/Los_Angeles",
immich_server            |     "inferredZone": true
immich_server            |   },
immich_server            |   "MediaModifyDate": {
immich_server            |     "_ctor": "ExifDateTime",
immich_server            |     "year": 2024,
immich_server            |     "month": 7,
immich_server            |     "day": 2,
immich_server            |     "hour": 9,
immich_server            |     "minute": 58,
immich_server            |     "second": 15,
immich_server            |     "tzoffsetMinutes": -420,
immich_server            |     "rawValue": "2024:07:02 16:58:15",
immich_server            |     "zoneName": "America/Los_Angeles",
immich_server            |     "inferredZone": true
immich_server            |   },
immich_server            |   "MediaTimeScale": 90000,
immich_server            |   "HandlerType": "NRT Metadata",
immich_server            |   "HandlerDescription": "MetaHandle",
immich_server            |   "MetaFormat": "mett",
immich_server            |   "MetaType": "application/microvideo-image-meta",
immich_server            |   "ImageSize": "1440x1080",
immich_server            |   "Megapixels": 1.6,
immich_server            |   "AvgBitrate": "13.7 Mbps",
immich_server            |   "Rotation": 90,
immich_server            |   "warnings": []
immich_server            | }
immich_server            | 
immich_server            | [Nest] 7  - 07/02/2024, 9:58:27 AM   DEBUG [Microservices:MapRepository] Request: <REDACTED>
immich_server            | [Nest] 7  - 07/02/2024, 9:58:27 AM VERBOSE [Microservices:MapRepository] Raw: {
immich_server            |   "id": 5339766,
immich_server            |   "name": "<REDACTED>",
immich_server            |   "longitude": <REDACTED>,
immich_server            |   "latitude": <REDACTED>,
immich_server            |   "countryCode": "US",
immich_server            |   "admin1Code": "<REDACTED>",
immich_server            |   "admin2Code": "<REDACTED>",
immich_server            |   "admin1Name": "<REDACTED>",
immich_server            |   "admin2Name": "<REDACTED>",
immich_server            |   "alternateNames": "<REDACTED>",
immich_server            |   "modificationDate": "2017-03-09"
immich_server            | }
immich_server            | [Nest] 16  - 07/02/2024, 9:58:27 AM   DEBUG [Api:LoggingInterceptor~zbtcnbh3] GET /api/jobs 200 17.55ms 76.21.24.156
immich_server            | [Nest] 7  - 07/02/2024, 9:58:28 AM     LOG [Microservices:MediaService] Successfully generated JPEG video preview for asset 13f56940-32c4-4311-a1a2-9adbe8c3ea86
immich_server            | [Nest] 7  - 07/02/2024, 9:58:28 AM     LOG [Microservices:MediaService] Started encoding video 13f56940-32c4-4311-a1a2-9adbe8c3ea86 {"inputOptions":[],"outputOptions":["-c:v h264_rkmpp","-c:a copy","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:2","-g 256","-v verbose","-vf scale=720:-2,format=yuv420p","-level 51","-rc_mode CQP","-qp_init 23"],"twoPass":false}
immich_server            | [Nest] 7  - 07/02/2024, 9:58:29 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/f7cecbb4-cf89-45d7-ba78-262c47c574f3/13/f5/13f56940-32c4-4311-a1a2-9adbe8c3ea86-preview.jpeg
immich_server            | [Nest] 7  - 07/02/2024, 9:58:31 AM     LOG [Microservices:MediaService] Successfully generated WEBP video thumbnail for asset 13f56940-32c4-4311-a1a2-9adbe8c3ea86
immich_server            | [Nest] 7  - 07/02/2024, 9:58:32 AM     LOG [Microservices:MediaService] Successfully encoded 13f56940-32c4-4311-a1a2-9adbe8c3ea86

Additional information

ffmpeg commands with and without hardware encoding:

Without: root 1950964 204 3.3 1103256 275352 ? SNl 10:06 0:06 /usr/bin/ffmpeg -i upload/upload/f7cecbb4-cf89-45d7-ba78-262c47c574f3/d9/64/d964c87c-76f2-436d-a8e6-d46a6f9bb10c.mp4 -y -c:v h264 -c:a copy -movflags faststart -fps_mode passthrough -map 0:0 -map 0:2 -v verbose -vf scale=720:-2,format=yuv420p -preset medium -crf 23 upload/encoded-video/f7cecbb4-cf89-45d7-ba78-262c47c574f3/0e/22/0e225248-9912-4c35-9113-4db64f671ee8.mp4

With: root 1951502 198 1.1 809368 97348 ? SNl 10:08 0:03 /usr/bin/ffmpeg -i upload/upload/f7cecbb4-cf89-45d7-ba78-262c47c574f3/d9/64/d964c87c-76f2-436d-a8e6-d46a6f9bb10c.mp4 -y -c:v h264_rkmpp -c:a copy -movflags faststart -fps_mode passthrough -map 0:0 -map 0:2 -g 256 -v verbose -vf scale=720:-2,format=yuv420p -level 51 -rc_mode CQP -qp_init 23 upload/encoded-video/f7cecbb4-cf89-45d7-ba78-262c47c574f3/0e/22/0e225248-9912-4c35-9113-4db64f671ee8.mp4

kaaku3 commented 4 months ago

I have the same bug in 1.108.0

kaaku3 commented 3 months ago

If i disable hdr tone mapping i do no have this issue, but there are significant artifacts to make hardware encode unusable.

mertalev commented 3 weeks ago

Can you try re-transcoding with 1.118.2 and clearing your browser cache to make it show the new transcode? We upgraded FFmpeg, so this might be fixed.

kaaku3 commented 3 weeks ago

After update to 1.118.2, I no longer seem to be getting green lines.

Seems to be fixed.

Possibly an unrelated bug, but the encoded video sometimes has severe artefacts and playback loops the first few frames of the video. It's inconsistent in reproducing, but I only noticed the issue using mobius tone-mapping and haven't noticed so far with hayble.