ARW files have broken previews in docker and Immich 1.61.0 run in Unraid.

[martihnus]

Hi, I am running Immich in docker via Unraid. Postgre and Redis are external to the Immich docker. Regenerated thumbnails multiple times, 0 active, 0 waiting.

Uploaded the .ARW via the web, the preview is broken on the Android app and on the web When I upload via the Android app, the preview shows up correctly on t he Android app but not on the web.

Does anyone have the same issue?

[martihnus]

error.log

[uhthomas]

1.61.0 introduced support for .ARW files (https://github.com/immich-app/immich/pull/2668). I can take a look later. The error log you provided doesn't really seem to have anything useful. It looks like just an nginx log. If you could upload a log from the server itself, that would be really helpful.

[uhthomas]

Ah. This image removes libvips-dev. This is a required runtime dependency.

https://github.com/imagegenius/docker-immich/blob/a372f56033242ddb2cbee32f31b8c83012f36407/Dockerfile

[martihnus]

1.61.0 introduced support for .ARW files (immich-app/immich#2668). I can take a look later. The error log you provided doesn't really seem to have anything useful. It looks like just an nginx log. If you could upload a log from the server itself, that would be really helpful.

I am sorry, here is log from Immich docker. log.log

[uhthomas]

Hmm, I do see a lot of missing files.

magick2vips: unable to read file \"/photos/library/dd249157-1c78-46ef-bb9e-4ab4e583bf16/2022/2022-09-07/sony_a7s_iii_01.arw

magick2vips: unable to read file \"/photos/library/dd249157-1c78-46ef-bb9e-4ab4e583bf16/2023/2023-04-30/DSC00650.arw

And for some jpegs too.

Unable to run job handler: Error: Input file is missing: /photos/library/dd249157-1c78-46ef-bb9e-4ab4e583bf16/2023/2023-04-18/IMG_20230418_162333.jpg

Is there something strange going on with your storage?

[martihnus]

I hope not. :D RAW files are deleted files from Immich when I was testing the ARW support. Folders 2022 and 2023-04-30 don't exist. IMG_20230418_162333.jpg does exist and Immich doesn't have a thumbnail for it. I deleted it.

[uhthomas]

I also see:

Unable to run job handler: Error: Input file has corrupt header: magick2vips: libMagick error: unable to open image `/tmp/magick-a5TbeyporC-oElajHUGD9vrCECWTuq1j.ppm': No such file or directory

Could you try removing the line I highlighted in an earlier comment? It should help.

[uhthomas]

It seems like this image is missing quite a few runtime dependencies. Namely imagemagick-dev and vips-magick.

https://github.com/immich-app/immich/blob/af86712df3b093cc2a342cff3279833ba055d520/server/Dockerfile#L26

[hydazz]

We’ll get those missing dependencies added asap 😊

[martabal]

Hello @uhthomas , I'm working on it but it seems even when libvips-dev is kept, it can't convert the raw images to jpeg.

It seems like this image is missing quite a few runtime dependencies. Namely imagemagick-dev and vips-magick.

https://github.com/immich-app/immich/blob/af86712df3b093cc2a342cff3279833ba055d520/server/Dockerfile#L26

Ubuntu does not have an imagemagick-dev package and libvips package is supposed to have everything to support imagemagick (see https://packages.ubuntu.com/lunar/libvips42)

[uhthomas]

@martabal I submitted a similar issue to Debian for https://gitlab.alpinelinux.org/alpine/aports/-/issues/14957. You'll have to check if the Ubuntu imagemagick package is compiled correctly with libraw support, as I believe Debian isn't. I'll try and find the bug tracker link for it later.

[uhthomas]

This error:

Unable to run job handler: Error: Input file has corrupt header: magick2vips: libMagick error: unable to open image `/tmp/magick-a5TbeyporC-oElajHUGD9vrCECWTuq1j.ppm': No such file or directory

Is exactly the same as the one on my bug report. I imagine it's the same issue. I'll follow up on it and see if Debian can publish a fixed package. For an immediate fix, imagemagick will have to be compiled from source.

[martabal]

compiled with ./configure --with-raw ?

[uhthomas]

I don't think you need to do anything special when compiling. As long as libraw-dev is installed during compilation, imagemagick should detect it and enable support for it.

[martabal]

I compiled ImageMagick 6 & 7, and still the same error

[uhthomas]

Did you see that imagemagick successfully found libraw during compilation?

-------------------------------------------------------------
checking for libraw_r >= 0.14.8... yes

-------------------------------------------------------------

[martabal]

yes

[uhthomas]

Does imagemagick work if you try it manually? Like I showed in my alpine issue?

[martabal]

No, I get the same error as you had. I tried with this Dockerfile

# syntax=docker/dockerfile:1

FROM ghcr.io/imagegenius/baseimage-ubuntu:lunar

# set version label
ARG BUILD_DATE
ARG VERSION
ARG IMMICH_VERSION
LABEL build_version="ImageGenius Version:- ${VERSION} Build-date:- ${BUILD_DATE}"
LABEL maintainer="hydazz, martabal"

# environment settings
ENV TRANSFORMERS_CACHE="/config/machine-learning" \
  TYPESENSE_DATA_DIR="/config/typesense" \
  TYPESENSE_VERSION="0.24.1" \
  TYPESENSE_API_KEY="xyz" \
  TYPESENSE_HOST="127.0.0.1" \
  PUBLIC_IMMICH_SERVER_URL="http://127.0.0.1:3001" \
  IMMICH_MACHINE_LEARNING_URL="http://127.0.0.1:3003" \
  IMMICH_MEDIA_LOCATION="/photos" \
  MACHINE_LEARNING_CACHE_FOLDER="/config/machine-learning"

RUN \
  echo "**** install runtime packages ****" && \
  echo "deb [signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_18.x lunar main" >>/etc/apt/sources.list.d/node.list && \
  curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | gpg --dearmor | tee /usr/share/keyrings/nodesource.gpg >/dev/null && \
  apt-get update && \
  apt-get install --no-install-recommends -y \
    ffmpeg \
    g++ \
    libheif1 \
    libraw-dev \
    libvips \
    libvips-dev \
    make \
    nginx \
    nodejs \
    perl \
    python3-dev \
    python3-pip \
    python3-venv && \
  echo "**** download immich ****" && \
  mkdir -p \
    /tmp/immich && \
  if [ -z ${IMMICH_VERSION} ]; then \
    IMMICH_VERSION=$(curl -sL https://api.github.com/repos/immich-app/immich/releases/latest | \
      jq -r '.tag_name'); \
  fi && \
  curl -o \
    /tmp/immich.tar.gz -L \
    "https://github.com/immich-app/immich/archive/${IMMICH_VERSION}.tar.gz" && \
  tar xf \
    /tmp/immich.tar.gz -C \
    /tmp/immich --strip-components=1 && \
  echo "**** download typesense ****" && \
  mkdir -p \
    /app/typesense && \
  curl -o  \
    /tmp/typesense.tar.gz -L \
    https://dl.typesense.org/releases/${TYPESENSE_VERSION}/typesense-server-${TYPESENSE_VERSION}-linux-arm64.tar.gz && \
  tar -xf \
    /tmp/typesense.tar.gz -C \
    /app/typesense && \
  echo "**** build server ****" && \
  cd /tmp/immich/server && \
  npm ci && \
  npm run build && \
  npm prune --omit=dev --omit=optional && \
  npm link && \
  npm cache clean --force && \
  mkdir -p \
    /app/immich/server && \
  cp -a \
    package.json \
    package-lock.json \
    node_modules \
    dist \
    /app/immich/server && \
  echo "**** build web ****" && \
  cd /tmp/immich/web && \
  npm ci && \
  npm run build && \
  npm prune --omit=dev && \
  mkdir -p \
    /app/immich/web && \
  cp -a \
    package.json \
    package-lock.json \
    node_modules \
    build \
    static \
    /app/immich/web && \
  echo "**** build machine-learning ****" && \
  cd /tmp/immich/machine-learning && \
  pip install --break-system-packages -U --no-cache-dir poetry && \
  python3 -m venv /lsiopy && \
  poetry config installer.max-workers 10 && \
  poetry config virtualenvs.create false && \
  poetry config virtualenvs.path /lsiopy && \
  poetry install --sync --no-interaction --no-ansi --no-root --only main && \
  mkdir -p \
    /app/immich/machine-learning && \
  cp -a \
    app \
    /app/immich/machine-learning && \
  echo "**** cleanup ****" && \
  apt-get remove -y --purge \
    g++ \
    make \
    python3-dev && \
  apt-get autoremove -y --purge && \
  apt-get clean && \
  rm -rf \
    /tmp/* \
    /var/tmp/* \
    /var/lib/apt/lists/* \
    /root/.cache \
    /root/.npm \
    /etc/apt/sources.list.d/node.list \
    /usr/share/keyrings/nodesource.gpg

# copy local files
COPY root/ /

# environment settings
ENV NODE_ENV="production"

# ports and volumes
EXPOSE 8080
VOLUME /config /uploads

and these commands

apt-get update
apt install -y build-essential git libraw-dev automake autoconf libtool gcc make cmake pkg-config libvips-tools
cd /tmp
git clone https://github.com/ImageMagick/ImageMagick.git
cd ImageMagick
./configure
make -j
make install
ldconfig /usr/local/lib
identify -verbose /photos/library/admin/2022/2022-04-20/RPA00519.arw

output:

root@f0181b1df0ed:/tmp/ImageMagick-7.1.1# identify -verbose /photos/library/admin/2022/2022-04-20/RPA00519.arw 
identify-im6.q16: delegate failed `'ufraw-batch' --silent --create-id=also --out-type=png --out-depth=16 --output='%u.png' '%i'' @ error/delegate.c/InvokeDelegate/1966.
identify-im6.q16: unable to open image `/tmp/magick-Cg3CejOxtBfn2dVHMr6Amzz_Oj43siHq.ppm': No such file or directory @ error/blob.c/OpenBlob/2924.

[martabal]

With

./configure --with-modules --enable-file-type --with-quantum-depth=32 --with-jpeg=yes --with-png=yes --with-gif=yes --with-webp=yes --with-heic=yes --with-raw=yes --with-tiff=yes --with-openjp2 --with-freetype=yes --with-webp=yes --with-openexr=yes --with-gslib=yes --with-gif=yes --with-perl=yes --with-jxl=yes

I get a correct output for identify -verbose /photos/library/admin/2022/2022-04-20/RPA00519.arw

root@6144b10ef5cf:/tmp/ImageMagick# identify -verbose /photos/library/admin/2022/2022-04-20/RPA00519.arw
Image:
  Filename: /photos/library/admin/2022/2022-04-20/RPA00519.arw
  Permissions: rw-r--r--
  Format: ARW (Sony Alpha Raw Image Format)
  Class: DirectClass
  Geometry: 7040x4688+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianness: Undefined
  Depth: 16/32-bit
  Channels: 3.0
  Channel depth:
    Red: 32-bit
    Green: 32-bit
    Blue: 32-bit
  Channel statistics:
    Pixels: 33003520
    Red:
      min: 0  (0)
      max: 65535 (1)
      mean: 53278.889 (0.81298374)
      median: 53458 (0.81571679)
      standard deviation: 5836.3243 (0.089056601)
      kurtosis: 9.4848467
      skewness: -1.537474
      entropy: 0.92903451
    Green:
      min: 4670  (0.071259632)
      max: 65535 (1)
      mean: 51210.115 (0.78141627)
      median: 51294 (0.78269627)
      standard deviation: 5357.3498 (0.081747918)
      kurtosis: 11.291851
      skewness: -1.6086923
      entropy: 0.92028137
    Blue:
      min: 0  (0)
      max: 65535 (1)
      mean: 45381.347 (0.69247496)
      median: 45406 (0.69285115)
      standard deviation: 4811.9735 (0.073426009)
      kurtosis: 12.24011
      skewness: -1.6570424
      entropy: 0.89688449
  Image statistics:
    Overall:
      min: 0  (0)
      max: 65535 (1)
      mean: 49956.784 (0.76229166)
      median: 50052.667 (0.76375474)
      standard deviation: 5335.2159 (0.081410176)
      kurtosis: 5.0943089
      skewness: -0.8824561
      entropy: 0.91540012
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33,0.03)
    green primary: (0.3,0.6,0.1)
    blue primary: (0.15,0.06,0.79)
    white point: (0.3127,0.329,0.3583)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: black
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 7040x4688+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Undefined
  Orientation: Undefined
  Profiles:
    Profile-xmp: 4097 bytes
  Properties:
    date:create: 2023-06-18T20:18:37+00:00
    date:modify: 2023-06-18T13:39:10+00:00
    date:timestamp: 2023-06-18T20:22:36+00:00
    dng:camera.model.name: ILCE-7M4
    dng:create.date: 2022-04-20T19:12:53+00:00
    dng:exposure.time: 1/125
    dng:f.number: 1.8
    dng:focal.length: 85.0 mm
    dng:focal.length.in.35mm.format: 85 mm
    dng:gps.altitude: 0.0 m
    dng:gps.latitude: 0 deg 0' 0.00" N
    dng:gps.longitude: 0 deg 0' 0.00" W
    dng:iso.setting: 160
    dng:lens: 85.0-0.0mm f/1.8-0.0
    dng:lens.f.stops: 0.00
    dng:lens.type: 
    dng:make: Sony
    dng:max.aperture.at.max.focal: 0.0
    dng:max.aperture.at.min.focal: 1.8
    dng:max.aperture.value: 1.8
    dng:max.focal.length: 0.0 mm
    dng:min.focal.length: 85.0 mm
    dng:software: ILCE-7M4 v1.00
    dng:wb.rb.levels: 2663.000000 1449.000000 1024.000000 1024.000000
    signature: b96da67537dc4196e10ec1119b1c2d1c02f7ce9160e02f375b2019a67a18e7ae
    xmp:Rating: 0
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 35475456B
  Number pixels: 33003520
  Pixel cache type: Memory
  Pixels per second: 29042157P
  User time: 4.560u
  Elapsed time: 0:02.136
  Version: ImageMagick 7.1.1-12 (Beta) Q32-HDRI x86_64 11ffa6eb4:20230529 https://imagemagick.org

But an error with vips thumbnail /photos/library/admin/2022/2022-04-20/RPA00519.arw test.jpg 100

(vips:54623): VIPS-WARNING **: 20:23:00.947: Unknown field with tag 50341 (0xc4a5) encountered

(vips:54623): VIPS-WARNING **: 20:23:00.966: Unknown field with tag 50341 (0xc4a5) encountered
magick2vips: libMagick error: unable to open image `/tmp/magick-W_BRud3vYi1jP-yB85QWn0_J64V3DXS2.ppm': No such file or directory @ error/blob.c/OpenBlob/2924
magick2vips: unable to read file "/photos/library/admin/2022/2022-04-20/RPA00519.arw"

[martabal]

convert /photos/library/admin/2022/2022-04-20/RPA00519.arw /photos/new.jpg works

[martabal]

@uhthomas any ideas ?

[uhthomas]

Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1038637

I sent an email weeks ago, but it looks like the email client I've been using (mailspring) ate it (twice!).

[uhthomas]

@martabal Well, the good thing is it looks like we're making progress. Not sure why vips would be unhappy if imagemagick is fine directly.

[martabal]

With

./configure --with-modules --enable-file-type --with-quantum-depth=32 --with-jpeg=yes --with-png=yes --with-gif=yes --with-webp=yes --with-heic=yes --with-raw=yes --with-tiff=yes --with-openjp2 --with-freetype=yes --with-webp=yes --with-openexr=yes --with-gslib=yes --with-gif=yes --with-perl=yes --with-jxl=yes

I get a correct output for identify -verbose /photos/library/admin/2022/2022-04-20/RPA00519.arw

Mmmh, it seems you don't need it, re-tried with only ./configure and it worked, however I still have the same error. I re-compiled libvips too and it does not work.

[martabal]

Great news, it seems compiling libvips and imagemagick with ./configure --with-modules resolves this issue.

[martihnus]

The latest build works, regenerated previews, and it all shows up. :) Thank guys.

[martabal]

Thanks for helping to debug !