Unsupported image format reported, but image is fine

[pbogre]

I am trying to import a collection of 50000+ images, however in the process of generating previews, object detection etc. I am getting the following errors:

[2024-06-30 11:11:14.634]: extractor.image.preview error Could not calculate image preview of <file>: 
Error: Could not calculate image preview from <file> with size 1920: Error: Input file contains unsupported image format

I noticed that when adding items to the queue to be processed, they are all detected as having a size of 4KB: Calculating ids for 200 entries with 0.8MB of total size 71.6MB (1.1%).

However the images are fine and I checked them myself. They all have an appropriate file size (checked using du -h) and I scp'd them to my laptop (Mac) and I was able to view them perfectly fine.

This problem happens with JPG, MOV, MP4, CR2 file formats (all the ones present in my collection). Checking them with ffprobe on my laptops shows no warnings or errors.

However it does not happen for all files in my collection, as 8216 of them get this problem while all others are processed perfectly. Below you can find one of the files that has this problem. It might also be worth noting that all the files that get this error were imported from my camera on the same day.

---

Docker compose file:

version: "3.8"
services:
  api:
    restart: unless-stopped
    image: xemle/home-gallery-api-server
    environment:
      - BACKEND=node
    networks:
      - gallery
  gallery:
    restart: unless-stopped
    image: xemle/home-gallery
    ports:
      - 6001:3000
    volumes:
      - /mnt/nas/files/family-photos:/data/Family
      - /docker/data/home-gallery:/data
    environment:
      - GALLERY_API_SERVER=http://api:3000
      - GALLERY_OPEN_BROWSER=false
      - GALLERY_WATCH_POLL_INTERVAL=300
    networks:
      - gallery
    entrypoint:
      - node
      - /app/gallery.js
    command:
      - run
      - server
networks:
  gallery: {}

gallery.config.yml:

#
# HomeGallery configuration file
#
#
# Default configuration file is gallery.config.yml. JSON format is
# also supported (gallery.config.json)
#
# This is a comment, starting with hash tag and space ('# ')
# A default value starting with hash tag and variable ('#baseDir...')
#
# Directory value examples:
# - /absolute/directory
# - relative/directory/to/current/working/dir
# - ~/directory/in/your/home
# - ./relative/directory/to/configuration/file
#
# Variable replacements
#   baseDir: '~'
#   configDir: '{baseDir}/.config/home-gallery'
# configDir is replaced to '~/.config/home-gallery' and than to '$HOME/.config/home-gallery'
#
# Variables are overwritten by environment variables
#
# Variables are baseDir, configDir, configPrefix, cacheDir and dir in sources
# or environment variables GALLERY_BASE_DIR, GALLERY_CONFIG_DIR, GALLERY_CONFIG_PREFIX, GALLERY_CACHE_DIR

#
# General
#
#baseDir: '~'
#configDir: '{baseDir}/.config/home-gallery'
# file prefix for index, database and events
#configPrefix: ''
#cacheDir: '{baseDir}/.cache/home-gallery'

#
# Sources
#
# List of media source directories. These can be read only.
#
# All sources are used to build the gallery database. If you need
# different databases or gallery instances use different gallery
# configurations
sources:
  - dir: '/data/Family'
    #index: '{configDir}/{configPrefix}{basename(dir)}.idx'
    # excludes are using gitignore patterns
    #excludes:
      #- .DS_Store
      #- ._*
      #- '*.tmp'
      #- '*cache*'
    #excludeIfPresent: .galleryignore
    # excludeFromFile: '{configDir}/excludes'
    # Use maxFilesize to exclude big files such as videos to speedup initial setup
    # maxFilesize: 20M
    # If source directory/disk is offline/unmounted set it to true.
    # Offline sources require an index file. Previews and meta data
    # should be extracted first before marking a source offline
    #offline: false
    # Filename matcher for checksum recalculation
    # size-ctime-inode: this matcher should be used if possible, might
    #                   not work on windows
    # size-ctime: this matcher should be used if stable fs inodes are
    #             not available and might not work for fuse shares
    # size: this matcher should be used if you know what you are doing
    #matcher: size-ctime-inode

#
# Extractor settings
#
extractor:
  # excludes:
  #   - *.xmp
  #   - *.svg
  # excludeFromFile: '{configDir}/excludes'
  # Preview image settings
  image:
    previewSizes: [1920, 800, 128]
    previewQuality: 80
  #video:
  #  previewSize: 720 # used as preview size and preview filename
  #  ext: mp4 # used as target video container and preview filename
  #  scale: -2:'min(720,ih)' # overwrites previewSize. E.g. -2:'min(ih,max(720,min(1080,ih*.5)))' for 720p <= height*0.5 <= 1080p or height, if height < 720
  #  frameRate: 30
  #  maxVideoBitRate: 4000 # in KB
  #  videoEncoder: libx264
  #  preset: slow
  #  profile: baseline
  #  level: '3.0'
  #  addFfmpegArgs: [...] # Additional ffmpeg args
  #  customFfmpegArgs: [...] # custom ffmpeg args for video conversion, replaces all other settings such previewSize, videoEncoder, addFfmpegArgs, ...
  # Api Server is used for image similarity, object and face detection
  apiServer:
  #  url: https://api.home-gallery.org
  #  timeout: 30
  #  concurrent: 5
    disable:
      - similarDetection
      - faceDetection
  #geoReverse:
  #  url: https://nominatim.openstreetmap.org
  #   Preffered address language of geo code reverse lookups
  #   addressLanguage: [de, en-US] # for multiple languages
  #  addressLanguage: [en, de]
  # excludes are using gitignore patterns
  #excludes: []
  # Use native system commands. It is recoomended on armv6 and arm7 platform
  # since the npm packages do not provide binaries for these platforms
  #useNative:
  #  - vipsthumbnail # use libvips to resize images
  #  - convert # use ImageMagick to resize images
  #  - ffprobe
  #  - ffmpeg

#
# Storage and files configuration
#
#storage:
  #dir: '{cacheDir}/storage'
#database:
  #file: '{configDir}/{configPrefix}database.db'
  # The full database creation is memory consuming. The default value is 2048 MB.
  # On memory errors or for larger photo collections (>50000 images) 4096 MB is recommended
  # Use less memory for smaller devices (e.g. Raspberry Pi Zero), e.g. 512 MB.
  # Node's default memory limit is 512 MB (changeable via --max-old-space-size node arg)
  maxMemory: 4096
  # excludes:
  #   - *.xmp
  #   - *.svg
  # excludeFromFile: '{configDir}/excludes'
  # supportedTypes
  #   - image
  #   - rawImage
  #   - video
#events:
  #file: '{configDir}/{configPrefix}events.db'

#
# Server configuration
#
#server:
  #port: 3000
  #host: '0.0.0.0'
  # security configuration for https
  # key: '{configDir}/server.key'
  # cert: '{configDir}/server.crt'
  # Open browser when server starts
  #openBrowser: true
  # Enable basic authentication users and ip whitelist rules which require no authentication
  # First matching rule wins. Default rules are: allow: localhost and deny: all.
  # Hashed password can be generated via
  #   node -e "pw=process.argv[1]||'';sha1=require('crypto').createHash('sha1').update(pw).digest('base64');console.log('{SHA}'+sha1)" password
  #   {SHA}W6ph5Mm5Pz8GgiULbPgzG37mj9g=
  # auth:
  #  users:
  #    - username: password
  #    - username: '{SHA}W6ph5Mm5Pz8GgiULbPgzG37mj9g='
  #  rules:
  #    - allow: localhost
  #    - allow: '192.168/16'
  #    - deny: all
  #basePath: /
  #watchSources: true

#
# Webapp configuration
#
#webapp:
  # Disabled webapp features
  #disabled:
    #- edit
    #- serverEvents
    #- pwa
    #- offlineDatabase

logger:
  # Currently console and file loggers are available
  - type: console
    # Log level could be one of trace, debug, info, warn, error, fatal, silent
    level: info
    # format: json
  # File logger format is in newline delimited JSON. See http://ndjson.org
  - type: file
    # Log level could be one of trace, debug, info, warn, error, fatal, silent
    level: debug
    file: '{configDir}/{configPrefix}gallery.log'

[xemle]

I am trying to import a collection of 50000+ images, however in the process of generating previews, object detection etc. I am getting the following errors:

[2024-06-30 11:11:14.634]: extractor.image.preview error Could not calculate image preview of <file>: 
Error: Could not calculate image preview from <file> with size 1920: Error: Input file contains unsupported image format

I noticed that when adding items to the queue to be processed, they are all detected as having a size of 4KB: Calculating ids for 200 entries with 0.8MB of total size 71.6MB (1.1%).

200 images with only 0.8 MB sounds strange and also your reported 4KB. Are these meta files from mac in some dot directories?

However the images are fine and I checked them myself. They all have an appropriate file size (checked using du -h) and I scp'd them to my laptop (Mac) and I was able to view them perfectly fine.

This problem happens with JPG, MOV, MP4, CR2 file formats (all the ones present in my collection). Checking them with ffprobe on my laptops shows no warnings or errors.

However it does not happen for all files in my collection, as 8216 of them get this problem while all others are processed perfectly.

Currently I do not have a clear picture what can cause this issue. Did you also check the files inside the container that these files can be read? You can cross check the SHA1 sum of the file index produced from HomeGallery and the SHA1 on the running container.

So I am also not know if it is a problem reading the complete file or it is a problem of sharp lib resizing the image.

Below you can find one of the files that has this problem.

Would you mind to attach the file, so I can check it on my side?

It might also be worth noting that all the files that get this error were imported from my camera on the same day.

Did you run HomeGallery previously without issues and updated the container recently? I did a larger refactoring from CommenJS to ES modules in the code. All my tests passed but maybe there is a problem? These changes were pushed 3 days ago.

Docker compose file:

The compose file looks OK to me.

The gallery.config.yml looks also good except one thing but this should not be the problem

#database:
  maxMemory: 4096

It should be with enabled database key

 database:
   maxMemory: 4096

[pbogre]

200 images with only 0.8 MB sounds strange and also your reported 4KB. Are these meta files from mac in some dot directories?

Ah, I hadn't thought about this. Your intuition was correct, the logs from home-gallery were reporting files that had ._ prefixed to their name, which I hadn't noticed thinking it was something to do with the formatting of the logs, and because I never thought of listing the files using ls -a. Checking the content of these files shows they come from MacOS, probably some indexing thing. Sorry about the false alarm.

It should be with enabled database key

I hadn't caught that, thanks!

[xemle]

You are welcome.

The ._* files can be excluded in the source section (as stated in the config) so that the gallery ignores these files.