Generalize perspective correction

[Lawrence37]

The flexibility of the perspective correction tool is limited by the current implementation's assumptions about the image. I feel there are two assumptions in particular which limit the tool's usefulness.

The first is the angular field of view. The current perspective tool does not consider the image's FOV. I have found that the tool works best for images with a 35mm equivalent focal length of around 20mm. For other focal lengths significantly different from 20mm, corrected images may appear stretched or compressed along one dimension. The circular depressions in the tower below are circular in reality, but appear elliptical in the corrected image: (towers_85mm.tif)

The second assumption is that, for horizontal correction, the axis of rotation is parallel with the image's vertical axis. In practice, this could mean that horizontal correction works well only if the camera was pointed straight forward (not aiming up or down). This assumption often results in skew. This is the best I could do with the current tool: (towers_28mm.tif)

I played around with the source code to see if removing the two assumptions would improve results. I am happy to report that it does! Here is a proof-of-concept at work: The axis of rotation for horizontal correction is dependent on the value for Vertical. The Vertical bias value allows for vertical correction without affecting the axis of rotation. Field of view is the diagonal angle of view.

I've placed some of my test images (created in Blender) and pp3s here: https://filebin.net/rmcnvpp9oj4ptla6 Also included is a patch containing the modifications I made. I grant permission to integrate the code into RawTherapee.

I would love it if the perspective tool becomes generalized by removing the two assumptions I described. I am willing to contribute to the coding.

[Thanatomanic]

Nice work. Please make a PR with your modifications so that more people can test them!

[gaaned92]

I was wondering why you don't borrow and perhaps improve ART automatic perspective correction code that I think was itself borrowed from darktable.

[Lawrence37]

@gaaned92 There has been unresolved discussion about automatic/semiautomatic perspective correction in #4505. I may be wrong, but I think the code to perform perspective transformation (which I have been working on) will be mostly independent from code used to automatically determine the slider values.

[gaaned92]

@Lawrence37 I understand these are different features.

• automatic perspective correction: the ART implementation is very effective. Correction can be manually modified if needed.
• perspective transformation: the ART perspective transformation code seems devoid of the two defects you noticed above. Perhaps it is a good starting point.

Backward compatibility: in my opinion, it is much more important to have a tool giving the correct result. In that case backward compatibility is a secondary problem.

[Floessie]

Backward compatibility: in my opinion, it is much more important to have a tool giving the correct result. In that case backward compatibility is a secondary problem.

Those with a bunch of fine-tuned PP3s in their backups will have another opinion, I suppose. And "correct" may mean something different to them.

[Lawrence37]

I was wondering why you don't borrow and perhaps improve ART automatic perspective correction code that I think was itself borrowed from darktable.

What improvements do you have in mind? Perhaps change the horizontal/vertical units to degrees? This can be useful if the EXIF contains the camera's pitch.

• perspective transformation: the ART perspective transformation code seems devoid of the two defects you noticed above. Perhaps it is a good starting point.

darktable/ART considers the focal length, which fixes the FOV problem. The horizontal perspective, however, behaves like RT's version. darktable solves this problem with the skew parameter. It makes combined vertical-horizontal manual correction more difficult than in my proof-of-concept.

---

After trying to make my code backwards compatible, I've come to the conclusion that the FOV used by the existing perspective tool depends on the amount of correction applied. I'm 95% certain this is the case because I had to readjust the FOV parameter in my implementation to match the current tool's output whenever I changed the horizontal/vertical sliders. It's possible my implementation is wrong, but highly unlikely as it agrees with Blender and Hugin. The 5% uncertainty comes from my failed attempts to reverse-engineer the math behind the current tool.

The changing FOV poses a problem when both horizontal and vertical transformations are used and have different values. Any replacement of the current tool will require two FOVs to maintain backwards compatibility.

I will pause my work on this to consider how to move on from here. Some options:

1. Abandon this.
2. Implement the two FOV parameters.
3. Add an improved perspective tool (based on my version, darktable, etc.) which goes before the existing tool in the processing pipeline.
4. Provide a choice between the existing implementation (Simple perspective) and an improved one (Advanced perspective), similar to Sharpening Method.
5. Something else.

I'm leaning towards 4. Comments?

[Beep6581]

Those with a bunch of fine-tuned PP3s in their backups will have another opinion, I suppose.

As I am one of those, I will allow myself to an opinion. I would absolutely support this feature regardless if it broke my old PP3s, because correcting perspective manually when I photograph paintings is a very boring and repetitive task which leads to imperfect results, so despite my manual efforts the perspective in my fine-tuned PP3s is still a bit wonky.

What improvements do you have in mind?

I would like the software to detect straight edges - vertical and horizontal - and try to make angles straight, so that a painting photographed at an angle (to avoid glare), or a building, can be straightened with a click. See #4505

I see that this issue is not about automatic correction but about improving the current approach. I have not had a chance to test it yet, but the examples look great.

[gaaned92]

I second the position of @Beep6581 . A strong reason why I use ART is that the automatic perspective is very easy to use and brings good results and correction is much better than in RT. Sometimes it needs manual adjustement to get a realistic perspective. I don't mind if it breaks my old pp3. If you want to keep compatibility , the best way should be:

• use the old processing for old PP3 ( there is a version number I think)
• only display the new processing in GUI. If it is activated, deactivate the old one .

The horizontal perspective, however, behaves like RT's version. darktable solves this problem with the skew parameter. It makes combined vertical-horizontal manual correction more difficult than in my proof-of-concept.

I think it would be a good sarting point as it features:

• automatic detection of horizontals and verticals ( paintings, buildings, indoor..)
• automatic detection of crop factor and focal length
• three automatic corrections
• can avoid wild cropping due to auto fill if deactivating it
• auto crop function

There can be some weakness to correct:

• non optimal combined vertical/horizontal correction as you noticed
• fixed dimensions of canvas though it should increase with correction
• ...

I see that this issue is not about automatic correction but about improving the current approach

I really hope you include automatic correction in your objectives, as there seems DT/ART can provide a good basis.

[Lawrence37]

Currently, I have implemented option 4 (choice between old and new tools). It maintains backwards compatibility and defaults to the old tool. A good thing about this design is it hides the 12 (!) adjusters in the new version until the correction method is changed. I can do what @gaaned92 suggests if no one feels strongly about keeping the old method in the GUI. 6 adjusters do not deal with perspective correction directly, so I can put them in a collapsible panel if desired.

darktable's automatic correction algorithm is fairly generic. It can be adapted to any decent perspective correction model. I strongly prefer my model over darktable's because I think the skew parameter makes manual fine-tuning difficult. I have started modifying darktable/ART's algorithm and hope to have a working version soon.

[Beep6581]

I haven't had time to test yet, but hope to find some this week. @Lawrence37 is PR #5569 ready for merging?

[Lawrence37]

There are a few small modifications I'd like to make. I will change the adjuster limits to more sensible values and likely remove the scale adjuster. The scale is equivalent to applying scaling at the end of all lens/geometric transformations. I believe it is more appropriate to have a scale adjuster outside of the perspective tool. If anyone has objections to removing scaling, let me know. I also have a change which I will push soon. It sets the default values of the focal length and crop factor in the editor according to the image metadata.

[afr-e]

I agree that the current perspective correction isn't intuitive. FYI, recently, I noticed the choice of demosaicing method trips up the auto algorithm in dt (https://github.com/darktable-org/darktable/issues/4364).

[Lawrence37]

I have pushed the changes mentioned above.

@afr-e Here's what I think is going on. The demosaicing algorithm affects the detection of lines slightly. The small changes in the lines can cause the lines to be classified differently (converging verticals vs. converging horizontals vs. neither). A small number of misclassifications can throw off the fitting algorithm. It's an edge case in which each stage of the algorithm amplifies a problem. Fortunately, the issue doesn't seem to affect the RT auto perspective tool (at least, for the image you used) and darktable gives users control over the line detection. I'm not saying the algorithm shouldn't be improved though.

[afr-e]

I thought as much. Merely sharing info for reference.