search

darktable-org / darktable

darktable is an open source photography workflow application and raw developer
https://www.darktable.org
GNU General Public License v3.0
9.54k stars 1.13k forks source link

Panasonic DMC-G7 hardcoded CFA is no longer valid #6988

Closed Fowlet closed 3 years ago

Fowlet commented 3 years ago

When loading a RAW (.rw2) file from a Panasonic Lumix G7, the colors are wrong, seems like no green channel?

Example: https://drive.google.com/file/d/10f5CN9Mw_GmI_7aQSlfv1YI675IyLIzS/view?usp=sharing

The RAW file that produced this: https://drive.google.com/file/d/1H1kc8w09lrIp261RmtBw8FEfUKgkmmFJ/view?usp=sharing

Nilvus commented 3 years ago

Could you provide the XMP also that display that? Did you check white balance settings and which darktable version do you use?

Fowlet commented 3 years ago

XMP: https://drive.google.com/file/d/1OKDmriPmKpxEDfCVHFBaCHHzPnMSUPRv/view?usp=sharing

The above WB is "camera" preset. All WB settings all produce varying shades of magenta.

Version 3.2.1 on Windows. Also tested on Ubuntu, version 2.4.2-1 (i.e. the one apt installs by default on 18.04).

The files open fine in e.g. RawTherapee.

Nilvus commented 3 years ago

I can easily reproduce that, even in master and with new color calibration module that has a powerful white balance option. With that module (and with white balance, I can improve colors and have a more balance and remove pink but it's hard on that image to restore all parts. The G7 doesn't seem to have a full set of white balance support as you can see here: https://github.com/darktable-org/darktable/blob/master/src/external/wb_presets.c (check around line 8272 to see that the G7 has few lines for white balance supports as, other like the G5 or the GX8 I have, have many more lines). So I suspect that should be the cause. And if I'm not wrong, that would mean work to improve the white balance presets. But only someone with a G7 could do that, so it will be hard to have better support.

You could see on that article, how to work on improving and propose a better white balance support. This is not easy but not really hard and needs a colorchecker: https://encrypted.pcode.nl/blog/2010/06/28/darktable-camera-color-profiling/

Nilvus commented 3 years ago

@johnny-bit: I just see that you add on april a lot of finetuned wb presets for panasonic bodies. Do you have some idea about that issue and how hard is it to set a correct white balance?

johnny-bit commented 3 years ago

Not really. This more likelly is some problem with the raw itself, because the error is present in dt 3.0.x, 3.2.1 as well as current master...

Maybe it's more for @LebedevRI ?

LebedevRI commented 3 years ago

@Fowlet was that raw file copied directly from camera's flash card, as a file, and it wasn't modified by any software?

chhil commented 3 years ago

I tried some random G7 raws from https://www.photographyblog.com/reviews/panasonic_lumix_dmc_g7_review/sample_images with current master. Did not see the problem.

chhil commented 3 years ago

I compared the exif info of a g7 file off the web and the one shared here.

The CFA pattern seems to be reversed.

Left OP image data , right internet image data

image

chhil commented 3 years ago

Since I was bored, I tried changing the CFA pattern in the raw....didn't help :(

Find group for CFA pattern (IFD0)
F:\portableapps\exiftoolgui>exiftool "-cfapattern*" -G1 P1000571.RW2
[IFD0]          CFA Pattern                     : [Green,Blue][Red,Green]

Update the group:cfapttern to reverse colors)
F:\portableapps\exiftoolgui>exiftool -IFD0:cfapattern="[Blue,Green][Green,Red]"
P1000571.RW2
    1 image files updated
See if colors were reversed.
F:\portableapps\exiftoolgui>exiftool "-cfapattern*" -G1 P1000571.RW2
[IFD0]          CFA Pattern                     : [Blue,Green][Green,Red]
LebedevRI commented 3 years ago

The CFA pattern seems to be reversed.

Yep.

Since I was bored, I tried changing the CFA pattern in the raw....didn't help :(

That's because we don't read the CFA specified in the EXIF, which is why this issue is here :]

chhil commented 3 years ago

That's because we don't read the CFA specified in the EXIF, which is why this issue is here :]

In that case why do G7 raws from the web work?

LebedevRI commented 3 years ago

That's because we don't read the CFA specified in the EXIF, which is why this issue is here :]

In that case why do G7 raws from the web work?

The more appropriate question is, why @Fowlet's raw doesn't.

Fowlet commented 3 years ago

I'll try resetting the camera to factory settings and see how the RAWs come out. I did mess with the color space settings but the only options are sRGB or Adobe RGB, neither of which are exotic, right?

LebedevRI commented 3 years ago

That didn't answer my question.

Fowlet commented 3 years ago

Apologies, it was downloaded from the camera via USB. Only possible source of weirdness is I uploaded it to my home server and was opening it from there, but I did try downloading it and opening it from local also with the same result. The server protocol is Samba which I would be very surprised if it messed with the internals of files.

LebedevRI commented 3 years ago

Downloaded with what? Plain file manager?

Fowlet commented 3 years ago

Yes, in the linux case GNOME File Explorer, in the Windows case File Explorer.

New test: I have factory reset the camera, formatted the SD card, set the capture mode to RAW and taken a picture. I have then connected the camera to a windows laptop via USB, and used darktable 3.2.1 to open the file from the camera as a "removable drive". Now things are more purple than magenta, as you can see in the below link.

I have uploaded the jpg, RAW and a darktable screenshot to google drive using the web interface, opening the files for upload directly from the camera/removable storage. https://drive.google.com/drive/folders/1D2xb3UYxpGgsg3gPvq_gRH9YrIPv4rv1?usp=sharing

chhil commented 3 years ago

@Fowlet Can you try shooting one with setting whitebalance to anything else than auto, something like shade or incandescent?

Fowlet commented 3 years ago

Here is a folder with a picture taken with WB set to shade (or "cloud symbol" at least): https://drive.google.com/drive/folders/1Lf0uL1uiVSJGUH2FI1y2_WvFX7F4zjnC?usp=sharing

LebedevRI commented 3 years ago

Yes, in the linux case GNOME File Explorer, in the Windows case File Explorer.

OK then. Can you please contribute the original raw (https://drive.google.com/file/d/1H1kc8w09lrIp261RmtBw8FEfUKgkmmFJ/view?usp=sharing), or some similar affected image to https://raw.pixls.us/ please ?

LebedevRI commented 3 years ago

@Fowlet ping

LebedevRI commented 3 years ago

@Fowlet ping

LebedevRI commented 3 years ago

Unless @Fowlet resurfaces with an CC0 sample, this won't be fixed.

todd-prior commented 3 years ago

Since I was bored, I tried changing the CFA pattern in the raw....didn't help :(

Find group for CFA pattern (IFD0)
F:\portableapps\exiftoolgui>exiftool "-cfapattern*" -G1 P1000571.RW2
[IFD0]          CFA Pattern                     : [Green,Blue][Red,Green]

Update the group:cfapttern to reverse colors)
F:\portableapps\exiftoolgui>exiftool -IFD0:cfapattern="[Blue,Green][Green,Red]"
P1000571.RW2
    1 image files updated
See if colors were reversed.
F:\portableapps\exiftoolgui>exiftool "-cfapattern*" -G1 P1000571.RW2
[IFD0]          CFA Pattern                     : [Blue,Green][Green,Red]

Change it in rawspeed directory....cameras.xml and it fixes the issue..

LebedevRI commented 3 years ago

Yep, and will break all the cameras where the existing CFA was correct. I'd like to address this properly, but for that i need someone affected to contribute an appropriate sample to RPU..

todd-prior commented 3 years ago

I was just helping someone on the FB forum with the issue... so I thought I would mention it....seems like some images firmware or software v1 use the BGGR and newer ones or maybe is CFA hardware and if so some are made with different filters on the sensor are GBRG?? The later was version 2.2 of the panasonic software.?? Bit out of my league here but I can likely dig up the two images...RPU?? Rawspeed update page?? thx for responding

LebedevRI commented 3 years ago

@todd-prior assuming that the images are straight from camera, and were copied via a card reader and file manager (i.e. were not modified/copied via some manufacturers/etc program), just tell them to contribute one such image to https://raw.pixls.us (adding a link to this disscussion in the upload's comment field) I'll take it from there. Thanks!

todd-prior commented 3 years ago

Done 17-01-2021

Sent a copy of a currently supported one and the alternate version that gives the magenta image on first opening due to altered CFA configuration

Will do

Get Outlook for Android

From: Roman Lebedev notifications@github.com Sent: Sunday, January 17, 2021 4:39:01 PM To: darktable-org/darktable darktable@noreply.github.com Cc: Prior,Todd priort@mcmaster.ca; Mention mention@noreply.github.com Subject: Re: [darktable-org/darktable] Panasonic DMC-G7 hardcoded CFA is no longer valid (#6988)  

@todd-prior assuming that the images are straight from camera, and were copied via a card reader and file manager (i.e. were not modified/copied via some manufacturers/etc program), just tell them to contribute one such image to https://raw.pixls.us (adding a link to this disscussion in the upload's comment field) I'll take it from there. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

LebedevRI commented 3 years ago

Perfect, sample acquisition confirmed. Please forward my thanks.

todd-prior commented 3 years ago

Will do….

From: Roman Lebedev notifications@github.com Sent: Monday, January 18, 2021 8:04 AM To: darktable-org/darktable darktable@noreply.github.com Cc: Prior,Todd priort@mcmaster.ca; Mention mention@noreply.github.com Subject: Re: [darktable-org/darktable] Panasonic DMC-G7 hardcoded CFA is no longer valid (#6988)

Perfect, sample acquisition confirmed. Please forward my thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/darktable-org/darktable/issues/6988#issuecomment-762237825, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKS2ES4MQLMFIOREJOKJQ5LS2QWVDANCNFSM4T35WV3A.

Martin-Luft commented 3 years ago

Same problem here. All my RAW photos of my Panasonic Lumix G70 are pink :(

todd-prior commented 3 years ago

@Martin-Wegner Its an easy hack to fix it or have you done so??

Martin-Luft commented 3 years ago

@Martin-Wegner Its an easy hack to fix it or have you done so??

Yes, thanks, it works now with:

            <Color x="0" y="0">GREEN</Color>
            <Color x="1" y="0">BLUE</Color>
            <Color x="0" y="1">RED</Color>
            <Color x="1" y="1">GREEN</Color>
todd-prior commented 3 years ago

You got it perfect…..

From: Martin-Wegner notifications@github.com Sent: January 28, 2021 1:37 PM To: darktable-org/darktable darktable@noreply.github.com Cc: Prior,Todd priort@mcmaster.ca; Mention mention@noreply.github.com Subject: Re: [darktable-org/darktable] Panasonic DMC-G7 hardcoded CFA is no longer valid (#6988)

@Martin-Wegnerhttps://github.com/Martin-Wegner Its an easy hack to fix it or have you done so??

Yes, thanks, it works now with:

                   <Color x="0" y="0">GREEN</Color>

                   <Color x="1" y="0">BLUE</Color>

                   <Color x="0" y="1">RED</Color>

                   <Color x="1" y="1">GREEN</Color>

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/darktable-org/darktable/issues/6988#issuecomment-769290505, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKS2ESZMTQH5PRUPXNU7I2LS4GVFZANCNFSM4T35WV3A.

codeman38 commented 3 years ago

Just got a Lumix G7 a couple weeks ago and encountered this same issue. I can also confirm that the XML change from @Martin-Wegner fixes the colors for me.

github-actions[bot] commented 3 years ago

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

LebedevRI commented 3 years ago

My apologies for completely dragging this out :( I seem to have completely dropped the ball on camera support for the last few months.