Additional filetype support (tiff, raw, etc.)

[sushimustwrite]

Exactly what it says on the tin. Support for more file types. Tiff and Raw are what I have in mind, but this issue is to make sure it's documented.

Video support is already covered in #616.

[hfiguiere]

Adding raw, forget about it. TIFF shouldn't be hard to allow.

[hfiguiere]

If I try to upload a tif file, I just get an error. Not very helpful to the end user.

PNG works though.

[httpdispatch]

Isn't it possible to add raw support to at least ImageImageMagick.php module? It looks like this lib supports raw conversion

[httpdispatch]

Looks like GraphicsMagic also may support raw files

GraphicsMagick requires 'dcraw' from http://www.cybercom.net/~dcoffin/dcraw/ to read raw images from digital cameras. Dcraw is invoked automatically when used to read files using a common RAW file format extension.

[httpdispatch]

I've successfully uploaded CRW photo after installing ufraw and dcraw and adjusting ImageImageMagick.php image format and removing mime type filter

->setImageFormat( 'jpg' );

Thumbnails size seems to be invalid also download is not working. But at least we know imagemagic can convert it

[hfiguiere]

you had to install a bunch of things that a hosting server will likely not have.

also what's the point of shooting raw if you do that.

[hfiguiere]

BTW the "->setImageFormat( 'jpg' );" is part of bug #1289.

[httpdispatch]

you had to install a bunch of things that a hosting server will likely not have.

I think that people who really need raw support will find hosting where they can install additional libraries. Also i think it is not a problem to install that libs for hosted version and offer raw support as a pro feature for example.

also what's the point of shooting raw if you do that.

I don't shoot in raw but pro photographers do it. From what i know this allows to adjust white balance and compensate exposure after the photo is done. Maybe some other stuff. I am not a pro photographer so don't really know.

BTW the "->setImageFormat( 'jpg' );"

Thanks, didn't know that. Just found raw conversion example on the stackoverflow and without that line conversion was not working.

[hfiguiere]

I think that people who really raw support know what they are doing and don't expect the hosting to support it. Otherwise they'd be shooting JPEG.

(btw I shoot raw, I'm not pro)

[httpdispatch]

@hfiguiere i think there are a lot of professionals who wants their raw backups. Also other photo storing services such as picturelife or flickr may support raw.

[hfiguiere]

Flickr doesn't.

[skomorokh]

RAW photos convert differently depending on what software you use to process them. Typically one would review and tweak exposure, wb, etc. While theoretically this might be encoded in an XMP or similar and passed through to a converter on the trovebox host, it is very unlikely it would render the image the same way as whichever package the user originally used to decide on the settings.

It might be useful to provide an alternate file to store alongside the full-size jpg/png. Perhaps as simple as considering any file uploaded in the same batch with the same name but with a different extension to be associated with the image? I'd not try and pick an extension to consider raw as output comes as ARW/CRW/CR2/RAW and many more.

Instead of just "original", there would instead be "original resolution" and, for those that have it, "source file".

[rufuspollock]

+1 for cr2/crw/raw support. I'd understood that the original file you provide does get stored in whatever is your selected storage so I'd imagine this would be easy to add ...

[jmathai]

@rgrp The RAW files would just be attached alongside their JPG files, right? I wonder if this could be implemented as a plugin as opposed to including it in the core. It'd result in less code and be faster to implement. The missing part of plugins is the ability to store data in the database as opposed to a configuration file.

[skomorokh]

Maybe there could be an object for each image called plugindata that gets serialised to JSON or whatever and has a key for each plugin?

Haven't really looked into the codebase but wouldn't the plugins also needs some hooks into the upload form to receive the extra files / metadata in the same place?

[jmathai]

@skomorokh I think a general plugin database would make more sense since a plugin might not be something that applies to specific photos. But in the general database you'd have the ability to link data to a specific photo by ID.

The hooks for the plugin already exist....

[rufuspollock]

@jmathai the issue is when i did an upload recently all the RAW stuff just got ignored (I couldn't even select it for upload ...)

[jmathai]

@rgrp Did you have an idea on how uploading RAWs would work? I think with SmugMug (correct me if anyone knows differently) you upload the JPG and then can upload a RAW to "attach" to that photo. The big problem with uploading just a RAW file is that the conversion to JPG differs and I'm not familiar with a library which handles all of those (again, happy to be proven wrong).

[hfiguiere]

On 07/10/13 01:10 PM, Jaisen Mathai wrote:

The big problem with uploading just a RAW file is that the conversion to JPG differs and I'm not familiar with a library which handles all of those (again, happy to be proven wrong).

I'll be frank: if you shoot RAW and expect the gallery software to do the conversion for you, you are doing it wrong. Shoot JPEG, the result will likely be better.

I'd be ok with "attach original RAW" in the context of saving it along the finished product.

[httpdispatch]

@jmathai if you remember i've did experiments and uploaded raw. Then on the server it was converted to jpeg automatically via ImageMagic. Why don't you want to use this approach?

[jmathai]

@httpdispatch I don't remember that. I haven't spent much time prior to see what can convert raw files but ImageMagick on my laptop (OSX) can do the following. I wonder what percentage of raw photos this covers.

        A* RAW       rw+   Raw alpha samples
      ARW  DNG       r--   Sony Alpha Raw Image Format
        B* RAW       rw+   Raw blue samples
      BGR* BGR       rw+   Raw blue, green, and red samples
     BGRA* BGR       rw+   Raw blue, green, red, and alpha samples
        C* RAW       rw+   Raw cyan samples
     CMYK* CMYK      rw+   Raw cyan, magenta, yellow, and black samples
    CMYKA* CMYK      rw+   Raw cyan, magenta, yellow, black, and alpha samples
      CR2  DNG       r--   Canon Digital Camera Raw Image Format
      CRW  DNG       r--   Canon Digital Camera Raw Image Format
      DCR  DNG       r--   Kodak Digital Camera Raw Image File
      DDS* DDS       r--   Microsoft DirectDraw Surface
      ERF  DNG       r--   Epson RAW Format
        G* RAW       rw+   Raw green samples
     GRAY* GRAY      rw+   Raw gray samples
        K* RAW       rw+   Raw black samples
      K25  DNG       r--   Kodak Digital Camera Raw Image Format
      KDC  DNG       r--   Kodak Digital Camera Raw Image Format
        M* RAW       rw+   Raw magenta samples
      M4V  MPEG      rw+   Raw MPEG-4 Video
      MEF  DNG       r--   Mamiya Raw Image File
     MONO* MONO      rw-   Raw bi-level bitmap
      MRW  DNG       r--   Sony (Minolta) Raw Image File
      NEF  DNG       r--   Nikon Digital SLR Camera Raw Image File
      NRW  DNG       r--   Nikon Digital SLR Camera Raw Image File
        O* RAW       rw+   Raw opacity samples
      ORF  DNG       r--   Olympus Digital Camera Raw Image File
      PCT* PICT      rw-   Apple Macintosh QuickDraw/PICT
     PICT* PICT      rw-   Apple Macintosh QuickDraw/PICT
        R* RAW       rw+   Raw red samples
      RAF  DNG       r--   Fuji CCD-RAW Graphic File
      RGB* RGB       rw+   Raw red, green, and blue samples
     RGBA* RGB       rw+   Raw red, green, blue, and alpha samples
     RGBO* RGB       rw+   Raw red, green, blue, and opacity samples
      SR2  DNG       r--   Sony Raw Format 2
      SRF  DNG       r--   Sony Raw Format
      X3F  DNG       r--   Sigma Camera RAW Picture File
        Y* RAW       rw+   Raw yellow samples
    YCbCr* YCbCr     rw+   Raw Y, Cb, and Cr samples
   YCbCrA* YCbCr     rw+   Raw Y, Cb, Cr, and alpha samples

[httpdispatch]

@jmathai you can scroll the issue top and see shots i've uploaded. There i demonstrated raw support is working.

[jmathai]

@httpdispatch I think what I'd like to see are sample RAW files from a variety of cameras which we can use to test.

[httpdispatch]

Here your go http://www.rawsamples.ch/index_en.php

[jmathai]

@httpdispatch let's track RAW specific discussion over at #1400

[jmathai]

Looks like we have a couple options here.

1. Extract thumbnails from the RAW files. This doesn't appear to be feasible since (as seen below) many RAWs have very low resolution thumbnails and a thumbnail isn't guaranteed at all.
2. Convert the RAW to a jpeg using ufraw. I can't speak to the quality of the conversion but it does appear to output high resolution JPEGs. Example here (5MB jpeg from L1004432.DNG is at L1004432.jpg).

Below are embedded thumbnails from #1400. Notice that the largest "thumbnail" is 1.2MB. That's actually great because we will resize it down to 1280x1280 for all thumbnail generation. Thus a 5MB thumbnail doesn't offer much more than a 1.2MB thumbnail. But it looks like we have to extract the high res thumbnail and go with it.

user# for i in $(ls) ; do echo "$i"; exiv2 -pp $i; echo "\n"; done
DSC08039.ARW
Preview 1: image/jpeg, 160x120 pixels, 2014 bytes
Preview 2: image/jpeg, 1616x1080 pixels, 306047 bytes

IMG_1842.CR2
Preview 1: image/jpeg, 160x120 pixels, 14720 bytes
Preview 2: image/tiff, 668x432 pixels, 1731456 bytes
Preview 3: image/jpeg, 5184x3456 pixels, 2355512 bytes

IMG_1888.dng
Preview 1: image/tiff, 256x171 pixels, 131328 bytes
Preview 2: image/tiff, 1024x683 pixels, 111101 bytes

L1004432.DNG
Preview 1: image/tiff, 320x216 pixels, 207360 bytes

_IGP0832.DNG
Preview 1: image/tiff, 160x120 pixels, 57600 bytes
Preview 2: image/jpeg, 640x480 pixels, 34106 bytes
Preview 3: image/tiff, 4928x3264 pixels, 1213896 bytes

__file.nef
Preview 1: image/tiff, 160x120 pixels, 57600 bytes
Preview 2: image/jpeg, 564x372 pixels, 25222 bytes
Preview 3: image/jpeg, 3008x2000 pixels, 734520 bytes

[jmathai]

Is there a way to tell the difference between tiff and RAW? The mime types of both are image/tiff.

user# file --mime-type IMG_1888.dng
IMG_1888.dng: image/tiff

[jmathai]

Here's a commit (537f444b50321ee7dfbb0b9071dfa1a87264e92c) that adds support for RAW. Looking for feedback. It's got 2 external dependencies (ufraw and exiftool) because I couldn't find anything in PHP to do the following...

1. Get the dimensions of a RAW file. getimagesize and exif_read_data both use the same logic and as far as I can tell will return the dimensions of an embedded thumbnail.
2. Convert a RAW file to a predictable JPEG resolution that's consistently large enough (see my prior comments).

Here's the screenshot of a rendered JPEG and the download link pointing to a DNG file.

[patricksan]

Is there a way to make the API always return a JPEG version? I think for mobile we shouldn't worry about RAW.

[jmathai]

Thats how it works. Only the original will return the raw.

On Thursday, January 23, 2014, Patrick Santana notifications@github.com wrote:

Is there a way to make the API always return a JPEG version? I think for mobile we shouldn't worry about RAW.

— Reply to this email directly or view it on GitHubhttps://github.com/photo/frontend/issues/1270#issuecomment-33118651 .

-- Snet form my mobl phoone

[patricksan]

perfect.

[jmathai]

Feel free to use this link to validate that your RAWs get processed correctly - https://trvbx.co/55d5e1ea69

[jmathai]

We're using ufraw-batch (version 0.19.2) and it looks like older versions don't do a great job on some Canon RAW files. On ubuntu this version is available in 14.04 trusty. YMMV getting it on your platform.

Earlier versions work but you might notice that the colors are off.

sudo apt-get install ufraw

ufraw-batch --version
ufraw-batch 0.19.2
EXIV2 enabled.
JPEG enabled.
JPEG2000 (libjasper) disabled.
TIFF enabled.
PNG enabled.
FITS disabled.
ZIP enabled.
BZIP2 enabled.
LENSFUN enabled.

[httpdispatch]

I've noticed that some uploaded raw photos has null dimensions returned by get photos list API. @jmathai can you check?

[jmathai]

@httpdispatch Do you have an example API response? If so please open as a new issue.

[httpdispatch]

@jmathai opened #1443