Offset registration

[Fastie]

Using the current version of RegisterPhotos, the early version that had the "Try SIFT" option. I processed 266 pairs of VIS and NIR jpeg photos from the Lincoln Peak Vineyards. The registration on many pairs seems to be poor compared to results from Lee, NH (all tiffs). Results from both registration methods (SIFT and bUnwarpJ) were poor on some pairs. The output images seemed to have a ghost of certain features as if the registered image was displaced a few dozen pixels. The ghost was often more pronounced toward one corner of the output images and easiest to see in the NRG images. The images are of a vineyard with lots of parallel linear features (arbors).

[nedhorning]

I think the offset problem is related to an insufficient number and more importantly distribution of correspondence points. We can try to tweak the SIFT algorithm to find more points or look into better ways to find correspondence points for the image pairs. When the SIFT points are not well distributed it is better to use the affine transformation in the landmark correspondences plugin than the bUnwarpJ if possible since the affine transformation distorts the image much less than bUnwarpJ.

In an attempt to increase the chances of getting the SIFT and landmark correspondences option to work I implemented a loop that tries 5 times to find enough points before giving up. The new version is on GitHub. I will also experiment with another transformation that is similar to affine but doesn't use scaling (only translation and rotation. That adds one more constraint which would probably improve registration of pairs with poorly distributed points but might sacrifice quality for pair with a good distribution. Just thinking out loud but will see how it turns out.

[nedhorning]

I just uploaded a new version that provides an (undocumented) option to select rigid or affine translation if you are using the SIFT option. The option you select is recorded in the log file that is saved. Can someone test the Rigid option to see if the results are better in general. My limited results are promising. I'll add this option to the guide after getting feedback.

[Fastie]

I processed all the pairs from Lee, NH with the current macro and output tiff. I used the camera jpegs as input (earlier runs with the Lee, NH pairs input tiff). I selected Affine registration when using SIFT. Of the 117 pairs, only eight were registered with bUnwarpJ. This is similar to earlier runs with a previous macro -- most of these pairs register well with SIFT and Afine. I have noticed no ghosts or shadows in the output produced with SIFT. However, some of the eight pairs registered with bUnwarpJ have ghosts from displaced registration. Two of the bUnwarpJ registrations were poor.

Conclusion 1: The SIFT registration works just as well on jpegs as it does on tiffs for these image pairs. Conclusion 2: Affine registration works well for these image pairs -- rigid probably would not be an improvement. Conclusion 3: bUnwarpJ sometimes does not produce good registration, but in these runs it is being tried only on the tougher pairs that SIFT is unable to register.

[Fastie]

I tried the rigid option on 133 pairs from Lincoln Peak. In general, the results are the same as with affine. I have compared only 40 or so of the 133, and I found two examples that have better registration after selecting the rigid option with the new version. One of these had been previously registered with bUnwarpJ, and this time was registered with SIFT and was better. The other was registered with SIFT both times and was better with rigid than affine. Small versions of the NRG images from pair 312 and pair 335 from an earlier run and today's run (today with rigid selected) are at Dropbox:

\Dropbox\Balloon Mapping\Chris\LPVsample\OutputSample\LPV-0312_NRG_bUnwarpJ.jpg" \Dropbox\Balloon Mapping\Chris\LPVsample\OutputSample\LPV-0312_NRG_rigid.jpg" \Dropbox\Balloon Mapping\Chris\LPVsample\OutputSample\LPV-0335_NRG_affine.jpg" \Dropbox\Balloon Mapping\Chris\LPVsample\OutputSample\LPV-0335_NRG_rigid.jpg"

Many of the output images I examined still have ghostly displacement.

[Fastie]

I compared the rest of the output from two macro runs on 133 pairs from Lincoln Peak. Nineteen out of 133 pairs were registered conspicuously better with the current version of the macro with rigid selected compared to an earlier version which tried SIFT only once and used affine instead of rigid. Of the 19 pairs registered better, 12 of them were registered with bUnwarpJ during the earlier run and were registered with SIFT (rigid) during the second run. The other seven were registered with SIFT both times but with affine the first time and with rigid the second time. Many of the other 133 image pairs did not register well and showed displaced ghosts which are especially conspicuous in the NRG output image.

It is obvious that the poorly registered pairs are images dominated by parallel rows of grape arbors. These repeating patterns are probably confusing the algorithms. The image pairs with buildings, trees, and roads are generally registered well. So the vineyard is a very hard test of the macro. These results suggest that certain agricultural landscapes are not good subjects for dual camera mapping because registration is a challenge.

[nedhorning]

Thanks for all the test runs Chris - that is a lot of work but it's very helpful. If the rigid and affine transformation result are similar I would recommend using rigid. Since it doesn't us a scale transformation there is one less chance to introduce distortions when we have a poor set of correspondence points. The weak point in the chain is the collection of matching points. I expect the main difficulty is due to the drastic difference between the visible and near-IR images. This week I will start testing some other approaches that might help improve the similarity of these two data sets. I'm also researching other options for selecting points but that's been slow going.

[nedhorning]

I added another option to the macro that allows you to select a pre-processing method and updated the guide.

The option “Method to improve point selection” provides a pre-processing method that can be used to improve the ability of the SIFT algorithm to find corresponding points by making the visible and near-IR look more similar. This is experimental and the only option at this time is "target (g-b) source (g+b)" which subtracts the blue channel from the green channel of the target (visible) image and adds the green channel to the blue channel of the source (near-IR) image to create input images for the SIFT algorithm. This approach seems to work well for the camera pairs made using the information on the PLOTS web site. Other approaches can be added for other camera pairs if this algorithm isn't effective.

If you test this let me know how it works.