Thumbnails from Sony ILCE-6700 RAWs (ARW files) contain black borders

[Snowknight26]

The bug

The thumbnail conversion that's done for RAWs made by a Sony ILCE-6700 (Alpha 6700) adds black borders to the image, regardless of thumbnail size.

On top of that, the dimensions reported by immich are also incorrect.

immich:

Lightroom, which matches JPGs made from the RAW:

Digging deeper into it it seems like it could be be because immich is using an older version of libraw that maybe doesn't fully support them.

Example file: https://drive.google.com/file/d/1erUSAJzyDywxfsG_nTXjrMwSc8VItMUQ/view?usp=sharing

The OS that Immich Server is running on

Windows 10 22H2 (19045.4291)

Version of Immich Server

1.101.0

Version of Immich Mobile App

N/A

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - "Z:\\Photos:/mnt/media/photos:ro"
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - "Z:\\Photos:/mnt/media/photos:ro"
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Add an external library that contains the ARW in the description
2. Wait for immich to generate thumbnails for the photo
3. Navigate to Photos and view the photo thumbnail

Relevant log output

No response

Additional information

No response

[mertalev]

Thanks for the report and example image. Could you double-check the permissions of that link? I can't access it.

[mertalev]

FWIW I locally opened the two example images for this camera from https://raw.pixls.us and the compressed one displays with the weird black border while the lossless one has some vignetting in the corners. The same occurs in immich.

However, there's a new option as of 1.102.0 to use the embedded preview for RAW images and I can confirm it works well for this camera. I recommend enabling this setting and re-running thumbnail generation for images from this camera.

[Snowknight26]

Thanks for the report and example image. Could you double-check the permissions of that link? I can't access it.

Fixed.

However, there's a new option as of 1.102.0 to use the embedded preview for RAW images and I can confirm it works well for this camera.

Sweet, looking forward to it.

[Snowknight26]

v1.102.0 using the new preview option produces even worse results. I mean.. technically the black border is gone, but at what cost? 😛

Left is RAW, right is JPG exported from RAW.

[mertalev]

It looks like there's an issue with how the preview image is produced to begin with - it has no metadata and the image looks like this locally as well. There's not much for an image converter to work with there. I checked the other example images and the same is true there as well; they just happened to produce a more normal looking image. The colors on them are still off upon closer inspection.

The issue with the RAW conversion seems to be that there are two different kinds of dimensions listed in the metadata: ImageHeight and ExifImageHeight (same for width and size as well). The former is what we use (and what the image processing libraries presumably use), while the latter is what has the correct dimensions for the image. Fixing it in our metadata extraction is easy, but the issue in image processing should be handled upstream. I think this is probably a libvips issue: libraw is only used for decoding the source image, while generating the converted image is handled in libvips.

Edit: It actually does look to be a libraw issue since I can reproduce it with their dcraw_emu CLI.

[mertalev]

I tested with the latest snapshot of libraw and still had this issue. Are you okay with me using this link as an example image for the issue?

[mertalev]

Never mind! I was apparently still using the release version. The recent snapshot does actually fix this issue.

[mertalev]

Fixed via #10457