Support RAW files (DNG, NEF, CR2, etc..)

[trednx]

Hello Is it possible to add DNG support? Each DNG has embedded preview, so I think, it will be not very resource "hungry" for RPI to show this preview without real-time rendering :) Thanks for your wonderful app!

[bpatrik]

Hi,

I was looking into supporting raw photos a few month earlier and there was not any simple solution for that.

I'd like to add the support of raw photos eventually, but not sure when.

Patrik

[ch33zer]

I'd like this support for CR2 files as well.

Until this is supported my plan was to use exiftool and cron to periodically extract the embedded thumbnail and store it alongside the raw file. This obviously requires more storage and it would be better if pigallery supported it out of the box.

I have a few thoughts for implementing this: 1)I know that pigallery has a tmp directory, would it be possible to, during indexing, extract the thumbnail and store it in tmp then serve that in place of the actual file? 2) Alternatively, upon receiving a request for a raw file rather than serving the file itself could we extract the thumbnail and just return that without even storing it anywhere?

I'm sorry if these are ignorant suggestions, I haven't actually looked at the code in depth to see what's possible. If I find some time I may work on this.

[bpatrik]

My plans to have raw photos support would be:

1) remove the GM and Jimp based thumbnail generation (that would increase the complexity by a lot). Keep sharp as that is the fastest 2) Add extensions you would like to support with converting/extraction here: https://github.com/bpatrik/pigallery2/blob/b6f93ac5c4e2f9939ba2745f65f21da0dd3bfcc4/src/common/SupportedFormats.ts#L24 3) check which of them cannot be handled by sharp (check multiple versions of CR2, DNG files as some of them work out of the box, some of them not) 4) find a node module that supports CR2, DNG converting (preview extracting) (should be fast, run on multiple platforms and have the least external dependencies) 5) Implement the conversion / (preview extracting here): https://github.com/bpatrik/pigallery2/blob/master/src/backend/model/threading/PhotoWorker.ts PhotoWorker works with these inputs: https://github.com/bpatrik/pigallery2/blob/b6f93ac5c4e2f9939ba2745f65f21da0dd3bfcc4/src/backend/model/threading/PhotoWorker.ts#L46-L59 (This is the expected output image)

I wont have time for this in the near future, but feel free to jump in :)

[lyonzy]

Hey guys,

I had a bit of a crack at this and managed to get it working for ORFs (Olympus) - albeit with some hacks.

I used raw-preview which is a bit undermaintained but very simple (native interface to exiv2). I also tried exiv2node but I couldn't get it to build.

What was required was fairly simple:

• SupportedFormats change as above

• This in PhotoWorker:

  private static async getRawPreview(filename:string):Promise<Buffer> {
  // n.b. make sure you require sharp before raw-preview otherwise you'll get this:
  // https://github.com/lovell/sharp/issues/1428#issuecomment-432558430
  var rp = await RawPreview.getPreviewAndMetadata(filename);
  return new Buffer(rp.image.base64, 'base64');
  }

  • And this inside the worker for Sharp (didn't do the other ones at this stage):

    if (input.mediaPath.toLowerCase().endsWith('.orf')) {
    Logger.silly('[SharpRenderer] rendering photo from raw preview:' + input.mediaPath + ', size:' + input.size);
    image = sharp(await this.getRawPreview(input.mediaPath), {failOnError: false});
    } else {
    Logger.silly('[SharpRenderer] rendering photo:' + input.mediaPath + ', size:' + input.size);
    image = sharp(input.mediaPath, {failOnError: false});
    }

  • MetadataLoader needed to be updated to get metadata for the ORFs, otherwise they'll all appear squished square, and not rotated based on Orientation. Luckily for me ORFs are just TIFFs with a different header so I could just change the header and process them as TIFFs with ExifReader:

          if (fullPath.toLowerCase().endsWith('.orf')) {
              data[2] = 0x2a;
              data[3] = 0x00;
              var exifTags = ExifReader.load(data);
    
              exif = {
                  tags: {
                      // this is a different value in ExifReader
                      ISO: exifTags.ISOSpeedRatings.value
                  }
              };
    
              // pull all these out of the .value field, and remove a few unnecessary big ones
              for (var tag in exifTags) {
                if (tag != "PrintIM" && tag != "MakerNote" && tag != "UserComment") {
                    exif.tags[tag] = exifTags[tag].value;
                }
              }
          } else {
              exif = ExifParserFactory.create(data).parse();
          }

  • I also needed to add this because the EXIF data I had didn't have any of the dimensions MetadataWorker was looking for:

          else if (exif.tags.ImageWidth && exif.tags.ImageLength) {
            metadata.size = {width: exif.tags.ImageWidth, height: exif.tags.ImageLength};
          }

    The ORF/TIFF approach probably isn't suitable long-term so you might choose to use raw-loader (it gets metadata too) or even better you might be able to pull it out of the exif using ts-exif-parser or ExifReader. However this relies on:

  • knowing where find exif in all the raw formats you want to support
  • knowing how to get the previews out of that exif (I couldn't find anything in the ORF files that was obviously a JPEG - I assume it's in MakerNote somewhere)

Obviously you'd need to find a way of knowing which files to apply this to that isn't path.endsWith('.orf') :) I guess either a list of supported formats for the raw-preview library/exiv2 or another field in SupportedFormats should do the trick.

Also an interesting note is that I also managed to get RAWs working with gm - I had to install imagemagick in the container I was using and then tell gm to use it (subClass({imageMagick: true});). This doesn't solve the metadata problem though, and it was fairly slow because I believe it was rendering them rather than extracting the preview (although I should note I ran it on a fairly slow box and my ORFs were on a NAS).

Hope this helps :)

[bpatrik]

Lets make this bug about supporting raw files. (renaming ticket to reflect is)

[bpatrik]

In the meantime point 1. in https://github.com/bpatrik/pigallery2/issues/127#issuecomment-588778193 is done.

The app uses sharp. From a quick google search I found, that sharp does not supports raw files: https://github.com/lovell/sharp/issues/616 But it sues libvips in the background that can be somehow tweaked: https://github.com/libvips/libvips/issues/537 It would require a more in-depth look to figure out.

[jezikk82]

Any chance we can see this feature in near future ?

[bpatrik]

no, I do not think so, unless someone implements it

[bpatrik]

I did a search and it seems (without looking into details too much) that sharp (the used image manipulation lib) can support raw files if we build vips ourself.

If i understand correctly changing vips to use magicload will allow us to open some raw files.

Some links:

1. https://github.com/lovell/sharp/issues/2852
2. https://github.com/libvips/libvips/issues/255

If my understanding is correct only the following changes are needed:

1. Build vips in the docker container with magicload.
2. Add the raw photo extensions in the settings.

So no code change is needed. This will already give a good enough (but still beta) experience

For a full solution a significant thumbnail and photo refactoring isneeded, without it zooming wont work with raw photos.

Would be appreciated if someone could figure out the vips building part in the Docker file

[papayavice]

Well, first of all, I am no developer at all. Just reading through this thread and a couple of more it seems like this issue here is relevant, another app but it uses sharp as well.

If I understood it right, the solution is to build sharp from Alpine with libraw-dev vips-dev vips-magick. vips-magick instead of vips to get magickload.

This is the repo from @uhthomas explaning how he did it.

I can't do it myself beacuse I don't know how but I am happy to test it in case this is the way to go.

Sorry, I was meant to reply to @bpatrik with this comment.

[uhthomas]

Well, first of all, I am no developer at all. Just reading through this thread and a couple of more it seems like this issue here is relevant, another app but it uses sharp as well.

If I understood it right, the solution is to build sharp from Alpine with libraw-dev vips-dev vips-magick. vips-magick instead of vips to get magickload.

This is the repo from @uhthomas explaning how he did it.

I can't do it myself beacuse I don't know how but I am happy to test it in case this is the way to go.

Sorry, I was meant to reply to @bpatrik with this comment.

Good morning!

Please see this PR for what was required to enable raw image support in Immich. This is for alpine, and uses fixes which are present in newer versions.

I believe the Debian authors also tried to fixed their implementation of vips too, though I haven't tested it and an initial glance seemed like it wouldn't work correctly. You may have to build vips yourself for Debian as a consequence. Immich also does this for other reasons if you look at the new Dockerfiles. I would test it first as they claim it should work.

Hope this is helpful.

[martadinata666]

Assuming that libraw-dev vips-dev vips-magick part work like how vips-heif work then it should be straightforward https://github.com/bpatrik/pigallery2/blob/master/docker/alpine/Dockerfile.build L4 to build sharp with raw support, and L29 to add raw format dependencies.

Furthermore the debian image should read raw, fixing heif here seems also supporting raw support due to libvips-dev in debian pulled all kind of image format available. Kindly pull bpatrik/pigallery2:edge-debian-bullseye if you love to test it, thanks.

[papayavice]

Furthermore the debian image should read raw, fixing heif here seems also supporting raw support due to libvips-dev in debian pulled all kind of image format available. Kindly pull bpatrik/pigallery2:edge-debian-bullseye if you love to test it, thanks.

Not working apparently.

Just to be sure, this is my docker compose.

version: '3.9'
services:
  pigallery2:
    image: bpatrik/pigallery2:edge-debian-bullseye
    container_name: pigallery2
    environment:
      - NODE_ENV=production # set to 'debug' for full debug logging
      # - NODE_OPTIONS=--enable-source-maps # enable source map support on the backend  for development
    volumes:
      - "/pigallery2/config:/app/data/config" # CHANGE ME
      - "db-data:/app/data/db"
      - "/pigallery2/test:/app/data/images:ro" # CHANGE ME, ':ro' means read-only
      - "/pigallery2/tmp:/app/data/tmp" # CHANGE ME
    ports:
      - 8080:8080
    restart: always

volumes:
  db-data:

This is my config.json

"supportedFormats": [ "gif", "jpeg", "jpg", "jpe", "png", "webp", "svg", "arw"

This is the error:

pigallery2 | message: 'Error during generating thumbnail: /app/data/images/115217.ARW', pigallery2 | details: 'Error: Input file contains unsupported image format', pigallery2 | request: { method: '', url: '' } pigallery2 | }

Screenshot:

[papayavice]

By the way, it reads tags from the xmp file in raw files, but nothing else. Date is wrong (I assume is taking modified file date).

EDIT: I gues this is the reason

[uhthomas]

This is the Debian bug tracker link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1038637

You may want to test with the latest release of Debian, or Debian unstable to ensure it genuinely hasn't been resolved.

[evrial]

is there a simple one line test printing all available libvips formats? We need to move forward at least baby steps next step is printing all sharp formats

[uhthomas]

is there a simple one line test printing all available libvips formats? We need to move forward at least baby steps

next step is printing all sharp formats

Yes, see https://github.com/uhthomas/sharp3612/blob/main/index.js

[papayavice]

We need to move forward at least baby steps next step is printing all sharp formats

Yes, please. It is the only thing left to make pigaller2 perfect. Any testing I'm down to do it.

[bpatrik]

is there a simple one line test printing all available libvips formats? We need to move forward at least baby steps next step is printing all sharp formats

Yes, see https://github.com/uhthomas/sharp3612/blob/main/index.js

If you want to try to run some debug log without getting deep into to pigallery2's code, you can try extensions with the edge build:

Docs: https://github.com/bpatrik/pigallery2/tree/master/extension

tldr: have a package.json and server.js in config/extension/myext folder Then this line runs at every startup: https://github.com/bpatrik/pigallery2-sample-extension/blob/70a36f01df7241520baebcdebb8a641e96d4843f/server.ts#L46

if you change any config in runtime, that line runs again.

[papayavice]

Any news on this? It is so close...

[bpatrik]

unfortunately not. I got super busy until like sept.

[bpatrik]

I made some changes (had no time to test):

1. Made a code that can test some exotic file types and auto disable them if the platform does not supports them: https://github.com/bpatrik/pigallery2/commit/894d73fedecc2f9efe400c0cab9490572fd37ad1
2. Added dng and arw test in 2ba1e11
3. Make that code run during docker build time in the "diagnostics step" c098dc6
4. added some vips magic support alpine: 2ba1e11

Outcome during running the build action on github:

Alpine seems to be fine now with dng but not with arw:

debian seems to support dng. I think it should not, so that check might not work properly:

This check should run also when tests run for a PR, so you should see the result there if you change the docker file.