Many issues with HDRMerge 5 and Sony A7RII files

[engardeknave]

1. HDRMerge 5.0 produces files that are pink and green, and full of artifacts. This also occurs with files from the original Sony A7R. 4.5 doesn't have this issue.
2. HDRMerge produces files that are 8000px in length from input files that are 7952px in length. The extra pixels appear as a smear at the right edge. This only happens with files from the A7RII and not the original A7R. It occurs when using either 5.0 or 4.5.
3. Sony Uncompressed Raw files result in files that are mostly pink with artifacts, but different from the first problem. This occurs when using either 5.0 or 4.5.

The following zip file contains two 3x3.0EV brackets, one uncompressed and one compressed RAW from the A7RII: https://www.dropbox.com/s/4k35hgknayk8g61/Sony%20A7RII%20Brackets.zip?dl=1

[heckflosse]

@engardeknave I took a look at the uncompressed files issue:

When you start hdrmerge -vv you get some info about the files. Here's what I got:

ingo@ingo-VirtualBox ~/hdrmerge/build $ hdrmerge -vv
camXyz values computed from rgbCam.
 3.24048 -1.53715 -0.498536
 -0.969255 1.87599 0.041556
 0.055647 -0.204041 1.05731
DSC00772.ARW: 8000x5320 (8000x5320+0+0, by SONY ILCE-7RM2, 100ISO 1/1.6sec f8 EV:-6.67807
b4b4b4b4 RGBG, sat 16383, black 0, flip 0, wb: 2464 1024 1588 1024, cblack: 0 0 0 0
camXyz values computed from rgbCam.
 3.24048 -1.53715 -0.498536
 -0.969255 1.87599 0.041556
 0.055647 -0.204041 1.05731
DSC00771.ARW: 8000x5320 (8000x5320+0+0, by SONY ILCE-7RM2, 100ISO 1/13sec f8 EV:-9.70044
b4b4b4b4 RGBG, sat 16383, black 0, flip 0, wb: 2464 1024 1588 1024, cblack: 0 0 0 0
camXyz values computed from rgbCam.
 3.24048 -1.53715 -0.498536
 -0.969255 1.87599 0.041556
 0.055647 -0.204041 1.05731
DSC00773.ARW: 8000x5320 (8000x5320+0+0, by SONY ILCE-7RM2, 100ISO 1/0.2sec f8 EV:-3.67807
b4b4b4b4 RGBG, sat 4095, black 0, flip 0, wb: 2464 1024 1588 1024, cblack: 0 0 0 0
Load files: 0.483928 seconds

Take a look at the sat value: the darker files have sat value 16383, but the brightest file has sat value 4095. I guess this different sat values (white levels) confuse hdrmerge.

@iliasg Can you take a look at the first three raw files from the above link (the uncompressed ones)? Do you have an explanation for the different white levels?

[iliasg]

I am not able to dig in for the moment, but this sounds same as the problems we had with RT and ARW uncompressed files tagged as 12 bit while in fact they are 14 bit (WL 16383) but with 3/4 of levels empty.

[heckflosse]

@iliasg Was this problem solved in rt or by dcraw update? I don't remember :(

[iliasg]

@heckflosse Ingo, you supplied a patch for RT excluding ARW2-3 files from relying on exif bitdepth tag. http://rawtherapee.com/oldforum/viewtopic.php?f=1&t=6316 https://github.com/Beep6581/RawTherapee/commit/77cf129bf23e8cb7aff9048f67fe4e56f849f8fa I don't know if/how latest Dcraw / Libraw solve this .. In any case, HDRmerge needs to get updated with latest Libraw to gain support for many new models.

[heckflosse]

@iliasg Thanks to your hint I found the current code to handle the issue. To test whether dcraw handles this now, I disabled it and then it fails again. Seems it's not yet fixed in dcraw.

[iliasg]

Yes Dcraw9.27 still failes on 12bit uncompressed ARWs .. .. latest Libraw (HDRmerge's decoding engine) can handle them with no problems ..

[engardeknave]

In my opinion the the uncompressed RAWs aren't the most pressing issue. Nobody really uses them. It would be great to get HDRMerge working with regular compressed RAW files from the A7RII. It would also be great not to have to crop the stretched pixels off the right edge.

Let me know if you need any more files.

[heckflosse]

The Issue with compressed arw files is caused by libraw. We tried today using hdrmerge with libraw 0.17.2 and set custom white level to 4075. That worked fine.

Ingo

Edit: Here's the result of latest hdrmerge using custom white level 4095

[StrobFX]

how do you use the latest libraw? I have a sony A7RII and I just tried HDRmerge and I get artifacts and wrong color. I was getting the wrong size (8000) with luminance HDR too.

[heckflosse]

@StrobFX How to use latest libraw depends on your OS. On my Sabayon Laptop I installed libraw 0.17.2 with the package manager and build HDRMerge. On my Mint Virtual machine I'm still at an old libraw version which does not work correctly with files from Sony A7RII. On my windows machine I can't build hdrmerge at all.

Btw: I still got the 'wrong' size (8000) using libraw 0.17.2 iirc but RawTherapee cut it down to the correct size during import (though it would be better if libraw would do that).

Ingo

[StrobFX]

Ok thanks I have to build the software from the binary. It's still something I have never done. By any chance do you have any link to a tutorial showing step by step how to build a realease? I'm windows but planning to install Unix soon.

[heckflosse]

@StrobFX There are instructions how to make a windows version of HDRMerge on a Linux system (Cross compile) but I never tried that. Building a Linux version of HDRMerge on Linux is easier. I just configured it a week ago on my Sabayon Laptop and it did not take much time, but I didn't record the steps. Once correctly configured it's also easy and fast to stay up to date with latest HDRMerge code (latest libraw may be a different story). Feel free to ask me again when you installed a Linux.

Ingo

[Beep6581]

Ok thanks I have to build the software from the binary.

The other way round: you have to build a binary from source.

I wrote this script to automatically get the source code and build the latest git version of HDRMerge. It's for Linux, but you may still find it useful. I suppose the biggest problem with compiling HDRMerge in Windows would be getting the dependencies. Once that's done, compiling HDRMerge should be easy and similar to Linux.

#!/usr/bin/env bash
prog=hdrmerge
#branches=("master" "develop")
branches=("master")

pushd $HOME/programs || exit
printf "%b\n" "\E[1mCloning ${prog}\E[0m"
if ! [[ -d code-${prog} ]]; then
    git clone https://github.com/jcelaya/hdrmerge.git "code-${prog}" || exit
fi
cd "code-${prog}" || exit

for branch in "${branches[@]}"; do
    if [[ $1 != "-b" ]]; then
        printf "%b\n" "Checking out branch \E[1m${branch}\E[0m"
        git pull || exit
        git checkout "${branch}" || exit
    fi
    rm -rvf "$HOME/programs/${prog}-${branch}"
    mkdir -p "$HOME/programs/${prog}-${branch}"
    cd "$HOME/programs/${prog}-${branch}"
    printf "%b\n" "Compiling branch \E[1m${branch}\E[0m"
    cmake -DCMAKE_INSTALL_PREFIX="$HOME/programs/${prog}-${branch}" -DCMAKE_BUILD_TYPE=release -DCMAKE_C_FLAGS="-O2 -pipe" -DCMAKE_CXX_FLAGS="${CMAKE_C_FLAGS}" "$HOME/programs/code-${prog}"
    make -j8 && printf "%s\n" "" "To run HDRMerge-${branch} type:" "~/programs/${prog}-${branch}/hdrmerge" ""
done
popd

[engardeknave]

I guess I'd need to compile a new version to get that custom white balance option? I'm using the 0.5 release on Windows.

[heckflosse]

@engardeknave Yes, exactly. If you're on Linux, try the script from @Beep6581

[heckflosse]

@engardeknave I didn't read your comment correctly. Now I saw you're on windows. My bad. But @Beep6581 currently tries to cross compile a new windows version on Linux. We hope that will work.

Ingo

[ghost]

@engardeknave

Nobody really uses them.

Hey, I'm using them! :) Seriously, I shoot only in uncompressed raw on a7r II.

[Beep6581]

I tested 12-bit and 14-bit compressed and uncompressed ILCE-7RM2 photos, no issue found.

v0.5.0-43-g 23fde0a libraw-0.18.5