Support for Lumix DMC-GF1 RAW format

[GoogleCodeExporter]

What steps will reproduce the problem?
1. try to extract with example code some Metadata from rawfile
2. http://www.mavinci.de/download/test.RW2 
3. Camera: Panasonic Lumix DMC-GF1 

What is the expected output? What do you see instead?
no exception, but all metadata containers are empty...

What version of the product are you using? On what operating system?
metadata-extractor-2.5.0-RC2.jar
openSuse 11.4 64bit

Please provide any additional information below.
I was able to view the exif stuff via "digicam" (linux application)

THX!
-Marco

Original issue reported on code.google.com by marcomoe...@gmail.com on 8 Sep 2011 at 6:27

[GoogleCodeExporter]

Hi Marco,

Thank you for this bug report.

When I run the latest code, I see the following error:

    ***** PROCESSING: D:\Dev\MetadataExtractor\Problem Images\Issue35\test.RW2
    ERROR: Invalid Exif start - should have 0x2A at offset 8 in Exif header

However I'm able to open the file in Photoshop and see the Exif metadata in the 
Camera Raw application.  So obviously metadata-extractor doesn't do a good job 
of handling this file format... yet!

I will add support for this as soon as possible.

Do you permit me to include this image in the sample library for testing 
purposes?  If not, could you create an alternative image that you would allow?

Original comment by drewnoakes on 14 Nov 2011 at 11:55

• Changed title: Support for Lumix DMC-GF1 RAW format
• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Why did some of my pics save as RW2 files? I am confused and how do I fix them.

Original comment by teamash...@gmail.com on 23 Dec 2011 at 2:38

[GoogleCodeExporter]

Hello. I tried using this with Panasonic Lumix GF-3 and observed the same thing 
on 2.4.0-beta-1. All metadata containers are empty.

You can look at the image here
http://dl.dropbox.com/u/37960/Temp/P1020249.RW2

You can also include this in the sample library for testing purpose if you'd 
like to

Original comment by varo...@panusuwan.com on 29 Jan 2012 at 9:31

[GoogleCodeExporter]

I tried to just remove the LOC that throws the message. It is in 
extractIFD(...) line 151 on r114. At that point, I can get to the information I 
need (ExifSubIFDDirectory.TAG_DATETIME_ORIGINAL)

Original comment by varo...@panusuwan.com on 29 Jan 2012 at 9:51

[GoogleCodeExporter]

Thanks for the additional sample image.

You're right insofar as removing that check does allow the image to be 
processed.  It seems that this camera model writes invalid data into the TIFF 
header, using the value 0x55 instead of 0x2A as the specification requires 
(http://gvsoft.homedns.org/exif/exif-explanation.html#TiffHeader).

The camera also uses some non-standard tags in IFD0:

***** PROCESSING: D:\Dev\MetadataExtractor\Problem Images\Issue35\test.RW2
[Exif IFD0] Unknown tag (0x0001) = 48 51 49 48
[Exif IFD0] Unknown tag (0x0002) = 4060
[Exif IFD0] Unknown tag (0x0003) = 3016
[Exif IFD0] Unknown tag (0x0004) = 4
[Exif IFD0] Unknown tag (0x0005) = 8
[Exif IFD0] Unknown tag (0x0006) = 3004
[Exif IFD0] Unknown tag (0x0007) = 4008
[Exif IFD0] Unknown tag (0x0008) = 1
[Exif IFD0] Unknown tag (0x0009) = 3
[Exif IFD0] Unknown tag (0x000a) = 12
[Exif IFD0] Unknown tag (0x000b) = 34316
[Exif IFD0] Unknown tag (0x000d) = 1
[Exif IFD0] Unknown tag (0x000e) = 16383
[Exif IFD0] Unknown tag (0x000f) = 16383
[Exif IFD0] Unknown tag (0x0010) = 16383
[Exif IFD0] Unknown tag (0x0017) = 100
[Exif IFD0] Unknown tag (0x0018) = 0
[Exif IFD0] Unknown tag (0x0019) = 0
[Exif IFD0] Unknown tag (0x001a) = 0
[Exif IFD0] Unknown tag (0x001b) = [42 bytes]
[Exif IFD0] Unknown tag (0x001c) = 0
[Exif IFD0] Unknown tag (0x001d) = 0
[Exif IFD0] Unknown tag (0x001e) = 0
[Exif IFD0] Unknown tag (0x0024) = 485
[Exif IFD0] Unknown tag (0x0025) = 263
[Exif IFD0] Unknown tag (0x0026) = 404
[Exif IFD0] Unknown tag (0x0027) = [58 bytes]
[Exif IFD0] Unknown tag (0x0029) = [36 bytes]
[Exif IFD0] Unknown tag (0x002a) = [32 bytes]
[Exif IFD0] Unknown tag (0x002b) = [16 bytes]
[Exif IFD0] Unknown tag (0x002c) = [72 bytes]
[Exif IFD0] Unknown tag (0x002d) = 4
[Exif IFD0] Unknown tag (0x002e) = [665600 bytes]
[Exif IFD0] Make = Panasonic
[Exif IFD0] Model = DMC-GF1
[Exif IFD0] Unknown tag (0x0111) = -1
[Exif IFD0] Orientation = Top, left side (Horizontal / normal)
[Exif IFD0] Unknown tag (0x0116) = 3016
[Exif IFD0] Unknown tag (0x0117) = 0
[Exif IFD0] Unknown tag (0x0118) = 667648
[Exif IFD0] Unknown tag (0x0119) = [32 bytes]
[Exif IFD0] X Resolution = 2 dots per unit
[Exif IFD0] Y Resolution = null
ERROR: Invalid Exif start - should have 0x2A at offset 2 in Exif header, not 
0x55.
[Exif SubIFD] Exposure Time = 1/1000 sec
[Exif SubIFD] F-Number = F5
[Exif SubIFD] Exposure Program = Shutter priority
[Exif SubIFD] Exif Version = 2.21
[Exif SubIFD] Date/Time Original = 2010:11:05 15:19:26
[Exif SubIFD] Date/Time Digitized = 2010:11:05 15:19:26
[Exif SubIFD] Exposure Bias Value = 0 EV
[Exif SubIFD] Max Aperture Value = F1.7
[Exif SubIFD] Metering Mode = Multi-segment
[Exif SubIFD] Flash = Flash did not fire, auto
[Exif SubIFD] Focal Length = 20.0 mm
[Exif SubIFD] File Source = Digital Still Camera (DSC)

It seems that this non-standard TIFF header may be a hint that the IFD being 
processed is not Exif IFD0 but rather a special IFD with tags specific to 
Panasonic RAW files.  Exiftool produces some nice output:

ExifTool Version Number         : 8.77
File Name                       : P1020249.RW2
Directory                       : D:/Dev/MetadataExtractor/Problem 
Images/Issue35
File Size                       : 13 MB
File Modification Date/Time     : 2012:01:29 22:27:48+00:00
File Permissions                : rw-rw-rw-
File Type                       : RW2
MIME Type                       : image/x-panasonic-rw2
Exif Byte Order                 : Little-endian (Intel, II)
Panasonic Raw Version           : 0310
Sensor Width                    : 4088
Sensor Height                   : 2688
Sensor Top Border               : 4
Sensor Left Border              : 8
Image Height                    : 2676
Image Width                     : 4008
ISO                             : 160
WB Red Level                    : 529
WB Green Level                  : 265
WB Blue Level                   : 380
X Resolution                    : 180
Y Resolution                    : 180
Resolution Unit                 : inches
Software                        : Ver.1.0
Modify Date                     : 2011:12:18 01:08:12
Y Cb Cr Positioning             : Co-sited
Sensitivity Type                : Standard Output Sensitivity
Components Configuration        : Y, Cb, Cr, -
Compressed Bits Per Pixel       : 2
Light Source                    : Unknown
Image Quality                   : Raw
Firmware Version                : 0.1.0.0
White Balance                   : Auto
Focus Mode                      : Auto
AF Area Mode                    : Auto or Face Detect
Image Stabilization             : Off
Macro Mode                      : Off
Shooting Mode                   : Scenery
Audio                           : No
Data Dump                       : (Binary data 6152 bytes, use -b option to 
extract)
Flash Bias                      : 0
Internal Serial Number          : (F97) 2011:08:23 no. 0266
Panasonic Exif Version          : 0360
Color Effect                    : Off
Time Since Power On             : 00:00:05.23
Burst Mode                      : Off
Sequence Number                 : 0
Contrast Mode                   : Unknown (0x11)
Noise Reduction                 : Standard
Self Timer                      : Off
Rotation                        : Horizontal (normal)
AF Assist Lamp                  : Enabled but Not Used
Color Mode                      : Normal
Baby Age                        : (not set)
Optical Zoom Mode               : Standard
Conversion Lens                 : Off
Travel Day                      : n/a
Contrast                        : Normal
World Time Location             : Home
Text Stamp                      : Off
Program ISO                     : n/a
Advanced Scene Mode             : Auto
Faces Detected                  : 0
Saturation                      : Normal
Sharpness                       : Normal
Film Mode                       : n/a
WB Adjust AB                    : 0
WB Adjust GM                    : 0
Panasonic Image Width           : 4000
Panasonic Image Height          : 2672
AF Point Position               : 0.5 0.5
Num Face Positions              : 0
Lens Type                       : LUMIX G 14/F2.5
Lens Serial Number              : 01HG6061141R
Accessory Type                  : NO-ACCESSORY
Faces Recognized                : 0
Title                           : 
Baby Name                       : 
Location                        : 
Intelligent Resolution          : Low
Intelligent D-Range             : Off
Maker Note Version              : 0138
Scene Mode                      : Scenery
Flash Fired                     : No
Flashpix Version                : 0100
Color Space                     : sRGB
Exif Image Width                : 1920
Exif Image Height               : 1280
Interoperability Index          : R98 - DCF basic file (sRGB)
Interoperability Version        : 0100
Sensing Method                  : One-chip color area
Scene Type                      : Directly photographed
Custom Rendered                 : Normal
Exposure Mode                   : Auto
Digital Zoom Ratio              : 0
Focal Length In 35mm Format     : 29 mm
Scene Capture Type              : Standard
Gain Control                    : Low gain up
PrintIM Version                 : 0250
Compression                     : JPEG (old-style)
Thumbnail Offset                : 11776
Thumbnail Length                : 4841
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:2 (2 1)
Jpg From Raw                    : (Binary data 612352 bytes, use -b option to 
extract)
Make                            : Panasonic
Camera Model Name               : DMC-GF3
Strip Offsets                   : 4294967295
Orientation                     : Horizontal (normal)
Rows Per Strip                  : 2688
Strip Byte Counts               : 0
Raw Data Offset                 : 614400
Exposure Time                   : 1/800
F Number                        : 7.1
Exposure Program                : Landscape
Exif Version                    : 0230
Date/Time Original              : 2011:12:18 01:08:12
Create Date                     : 2011:12:18 01:08:12
Exposure Compensation           : 0
Max Aperture Value              : 2.5
Metering Mode                   : Multi-segment
Flash                           : Off, Did not fire
Focal Length                    : 14.0 mm
File Source                     : Digital Camera
Aperture                        : 7.1
Blue Balance                    : 1.433962
Image Size                      : 4008x2676
Lens ID                         : LUMIX G 14mm F2.5
Red Balance                     : 1.996226
Scale Factor To 35 mm Equivalent: 2.1
Shutter Speed                   : 1/800
Thumbnail Image                 : (Binary data 4841 bytes, use -b option to 
extract)
Circle Of Confusion             : 0.015 mm
Field Of View                   : 63.7 deg
Focal Length                    : 14.0 mm (35 mm equivalent: 29.0 mm)
Hyperfocal Distance             : 1.90 m
Light Value                     : 14.6

I would like to investigate this further before committing code changes, as 
this may require a bit of a rethink to how RAW images are processed.  I'd also 
like to try and collate a few other RAW images from other cameras to see 
whether there's a standard pattern at work that I don't know about.

Original comment by drewnoakes on 29 Jan 2012 at 11:15

• Changed state: Started

[GoogleCodeExporter]

Hi,

I have uploaded another RAW-Image (and jpeg) here (Sony Nex-5), as soone it is 
avaliable I will support you with samples of Sony Nex-7.
www.mavinci.de/download/DSC01447.ARW
www.mavinci.de/download/DSC01442.JPG

you can add it you your demo-data reposiroty.

THX for supporting us! I am realy looking forward to get it working.. :-)

-Marco

Original comment by marcomoe...@gmail.com on 2 Feb 2012 at 2:03

[GoogleCodeExporter]

Hi,

here you can download a RAW image from Sony Nex7: (you are allowed to put this 
in your testcase repository)

http://www.mavinci.de/download/DSC00555.ARW

how is the current state of RAW support?

THX

-marco

Original comment by marcomoe...@gmail.com on 15 May 2012 at 12:18

[GoogleCodeExporter]

the download link for nex5 images was not hightlighted in comment #6...
http://www.mavinci.de/download/DSC01447.ARW
http://www.mavinci.de/download/DSC01442.JPG

Original comment by marcomoe...@gmail.com on 15 May 2012 at 12:24

[GoogleCodeExporter]

@Marco, thanks for the images.  Is there any important difference between the 
two ARW files?  I will include the first one in the repository for now, as 
they're quite large.

Thanks very much for your contribution.

Original comment by drewnoakes on 16 May 2012 at 10:25

[GoogleCodeExporter]

Hi, both are not working, and they are from different but similar cameras. 
That's all I know. I think after it is working with one camera, we will see if 
it is also working with the other one. 

In the next days I could support you also with gx1 samples. 

Marco 

Original comment by marcomoe...@gmail.com on 16 May 2012 at 10:35

[GoogleCodeExporter]

@marcomoeller, I have now uploaded your sample images, thank you.  I wasn't 
able to add them under SVN, presumably due to the file size.  I converted the 
project to Git hosting and was able to add them as well.

Original comment by drewnoakes on 18 May 2012 at 8:51

[GoogleCodeExporter]

Original comment by drewnoakes on 20 May 2012 at 5:51

• Added labels: RawFiles

[GoogleCodeExporter]

Original comment by drewnoakes on 16 Oct 2012 at 4:23

• Added labels: Component-FileTypes, Motive-Coverage
• Removed labels: RawFiles

[GoogleCodeExporter]

Original comment by drewnoakes on 16 Oct 2012 at 4:25

• Added labels: Milestone-2.7.0, Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Related to http://code.google.com/p/metadata-extractor/issues/detail?id=43

Original comment by drewnoakes on 16 Oct 2012 at 4:25

[GoogleCodeExporter]

Version 2.6.3 no longer refuses to process ARW and RW2 files.

However I'm leaving this ticket open as I'd still like to add special support 
for these cameras' tags, and this will require some refactoring.

Currently the ExifReader class is really a TiffReader. I believe it should be 
split in two, such that the ExifReader uses an instance of TiffReader, passing 
it the data to process and some metadata describing expected tags, sub-IFD 
definitions, etc.

The different tag definitions could be provided for different RAW formats.

Original comment by drewnoakes on 28 Oct 2012 at 6:59

• Added labels: Component-RAW, Component-TIFF, Priority-Medium
• Removed labels: Milestone-2.7.0, Priority-High

[GoogleCodeExporter]

Thanks! I will test this with our next major release. For now I have a working 
workaround ;-)

BR
-marco

Original comment by marcomoe...@gmail.com on 30 Oct 2012 at 4:54

[GoogleCodeExporter]

An update on this issue -- I've extracted TIFF processing into its own class, 
TiffReader, which is invoked with an implementation of the TiffHandler 
interface. Currently the only implementation of that if the ExifTiffHandler, 
but it is a step towards creating specific TiffHandlers for other IFD 
arrangements.

Original comment by drewnoakes on 15 Jan 2013 at 8:22

[GoogleCodeExporter]

This issue has been migrated along with the project to GitHub:

https://github.com/drewnoakes/metadata-extractor

Original comment by drewnoakes on 19 Nov 2014 at 6:08