how to use an `image_align` transformation as `sfs` input?

[steo85it]

Hi! Is it possible to apply a transformation resulting from image_align (applied to projected images) to their original cameras (.cub), which then serve as input for sfs? One could read the translation from image_align output and then add it to a bundle_adjust output template (which can be read into sfs), but those transformations are probably given in different reference frames. Any suggestions? Thanks!

[oleg-alexandrov]

In theory that would be possible. Since an orthorectified image is obtained by projecting onto a DEM, one would need to transfer the "before align" and "after align" matching orthoimage pixels to the DEM surface, convert to ECEF, and find the transform that fits these best, which this time will be in ECEF, so can be applied to cameras.

This will not be a precise solution, unless the DEM is perfectly flat. And then the SfS solution may show registration artifacts.

I am not sure what you use the tools for, but it may be preferable to do an accurate 3D registration instead. Ideally the images would be bundle-adjusted, a 3D terrain created, then that one aligned to your DEM and the cameras transferred over (the doc has the details).

Otherwise one could create GCP with the gui tool by point-and-click, and use those to move the cameras. That is also documented on the stereo_gui page.

In short, what you suggest would need some code work, and I am a bit reluctant to do that for now as I am not sure if this is a valid use case. However, such a produced 3D ECEF transform analogous to the transform in image space could have other uses too so there could be some value in that.

Would any of the above work instead?

[steo85it]

Thanks for the reply! I am/would be using this step (aligning the "computed" and "measured" intensities produced by a "0 iterations" preliminary sfs run, and passing the resulting corrections to the cameras for the "real" sfs run) to replace the 12.9.5 Alignment to ground step suggested in the manual (using pc_align) for the sfs recovery of lunar surfaces using LRO NAC images. That step very often fails for me and this seems to be an efficient replacement to both refine bundle_adjust results AND get a correct geolocation of the NAC images to the apriori DEM. As such, any "point-and-click" option wouldn't work for me (too many images and locations to process). The other options could work, but they indeed don't seem as straightforward as applying a simple "unprojection" or similar transformations.

 Ideally the images would be bundle-adjusted, a 3D terrain created, then that one aligned to your DEM and the cameras transferred over (the doc has the details).

What section/tool are you referring to, here? (Just to check if it's something I already tested/tried or not)

[oleg-alexandrov]

If you have a stereo DEM produced from some of the bundle-adjusted images, the following should be the most reliable approach: https://stereopipeline.readthedocs.io/en/latest/sfs_usage.html#sfs-and-alignment-in-lola-s-coordinates (there's more background higher up). If you think it won't work, let me know.

[steo85it]

I don't, but I do usually have images pairs (LE/RE), so that I could generate one (or more) "local" stereo terrains (they would be smaller than the tiles that I'm processing), align them to the LOLA DEM and apply the resulting transformations. pc_align hasn't been very reliable in my experience, and I tried to replace it, but let's see. Thanks, I'll let you know!

[oleg-alexandrov]

When it comes to this:

aligning the "computed" and "measured" intensities produced by a "0 iterations" preliminary sfs run

I guess what is going on is the following. The "measured" intensity mirrors the projected LRO NAC image. The "computed one" mirrors the input DEM. These will be of course misaligned if the input image is not registered to the DEM. Now, the input DEM is usually low-resolution, so its "computed intensity" can look quite different from the "measured" one, even apart from misalignment. I guess image_align worked for you here, which suggests your input DEM is rather accurate. In general this may have robustness issues.

You seem to be doing a rather large job here based on what you say. This can take an outrageous amount of time, many weeks that is, especially towards the poles and/or if the illumination conditions are highly diverse. Please do keep in touch, also privately if necessary, as this is one heck of a problem, and the tools have issues scaling to big workflows.

[dpmayerUSGS]

I don't, but I do usually have images pairs (LE/RE) Recall that there's hardly any stereo angle between images from the Left and Right cameras, and the overlap is very narrow to begin with so you're unlikely to be able to extract enough terrain to align to the LOLA DEM (pc_align needs enough area to "grip").

---

From: Stefano Bertone @.> Sent: Friday, January 20, 2023 14:41 To: NeoGeographyToolkit/StereoPipeline @.> Cc: Subscribed @.***> Subject: [EXTERNAL] Re: [NeoGeographyToolkit/StereoPipeline] how to use an image_align transformation as sfs input? (Issue #391)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

I don't, but I do usually have images pairs (LE/RE), so that I could generate one (or more) "local" stereo terrains (they would be smaller than the tiles that I'm processing), align them to the LOLA DEM and apply the resulting transformations. pc_align hasn't been very reliable in my experience, but let's see. Thanks, I'll let you know!

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNeoGeographyToolkit%2FStereoPipeline%2Fissues%2F391%23issuecomment-1398984439&data=05%7C01%7Cdpmayer%40usgs.gov%7C292f69cc185a4ead0e0008dafb2f09be%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C638098476704119493%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=S%2FzoPsc9Ys0%2FbcuiUZizD%2FVWGQIQSKa1aK%2Bi2xENhLg%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGILD2RB4QW5HB2OE3G4MC3WTMA67ANCNFSM6AAAAAAUB4SIAQ&data=05%7C01%7Cdpmayer%40usgs.gov%7C292f69cc185a4ead0e0008dafb2f09be%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C638098476704119493%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GPbnFMGG%2FtZogWvyqzEGq40NbBWjpEX9xRLn9KgMqfo%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[oleg-alexandrov]

David, good point. One needs actual stereo pairs here. LRO NAC doesn't usually acquire thing in stereo mode, so those can be tricky to find (though bundle_adjust as of recently saves the convergence angle for all encountered image pairs, which may help identify some).

If @steo85it does think the image_align features he wants will help, meaning that it creates a plausible alignment for the data he makes, I will cook it up in day or two as that's little work and this was requested before too. Not sure about the best overall workflow for registration for SfS though; this is a tough one.

[steo85it]

I guess what is going on is the following. The "measured" intensity mirrors the projected LRO NAC image. The "computed one" mirrors the input DEM. These will be of course misaligned if the input image is not registered to the DEM. Now, the input DEM is usually low-resolution, so its "computed intensity" can look quite different from the "measured" one, even apart from misalignment. I guess image_align worked for you here, which suggests your input DEM is rather accurate. In general this may have robustness issues.

This is precisely what I'm (tentatively) doing, and indeed it is working pretty well as the reference LDEM I'm using is decently accurate and has a similar resolution as my intended product. I can easily apply those transformations to the projected NAC and check that the overlap to the rendered LDEM improves, the problem is how to transfer those "corrections/xy-shifts" back to the (not-map-projected) cameras as you tell me that a "simple un-projection" wouldn't work.

You seem to be doing a rather large job here based on what you say. This can take an outrageous amount of time, many weeks that is, especially towards the poles and/or if the illumination conditions are highly diverse. Please do keep in touch, also privately if necessary, as this is one heck of a problem, and the tools have issues scaling to big workflows.

It is rather extensive, yes, although I do have a decently efficient pipeline already in place and a more or less functional interface to the ASP tools. My main issue is currently the co-registration of the images to the apriori DEM (and eventually refining bundle_adjust corrections, but that's "minor"). If you think that you can "easily" code such a feature, that would be great, thanks! And yes, for sure I will keep in touch, thanks (@rbeyer knows what I'm dealing with and is involved in the project, too)!

Recall that there's hardly any stereo angle between images from the Left and Right cameras, and the overlap is very narrow to begin with so you're unlikely to be able to extract enough terrain to align to the LOLA DEM (pc_align needs enough area to "grip").

ok, thanks for the remark! Then that's not really a convenient way to go for me...

[oleg-alexandrov]

I'll cook up the logic for transferring the 2D alignment between two georeferenced images to a 3D alignment relative to the planet, assuming that the DEMs used to make the georeference images are known. This is a nifty feature anyway, with other applications too. I'll confirm when it is done; it should be in the build by tomorrow morning.

Once your images are in the ballpark and mapprojection shows only small misregistration errors, the SfS doc suggests using the bundle_adjust --heights-from-dem option (with appropriate weight and threshold) to tighten the registration. This has worked very well for me for such problems if there are plenty of tie points among images with various illumination conditions.

[steo85it]

Thanks, that would be great. And good idea about leveraging the --heights-from-dem option: I'll try that step more carefully, too!

[oleg-alexandrov]

The functionality made it to the latest build at https://github.com/NeoGeographyToolkit/StereoPipeline/releases. The documentation is at https://stereopipeline.readthedocs.io/en/latest/tools/image_align.html#determination-of-ecef-transform. I fetched that build and I am getting correct results for a testcase with a known answer.

Yes, bundle adjustment can be tricky even after one is in the ballpark. Apart from insufficient ties among images with different illumination, I ran into issues with low-res images with wide footprint, and also with jitter. If say 3/4 of the images work out, the rest can just be thrown out rather than trying to shoehorn them.

If running into issues, one can also temporarily scale down the problem to something smaller but still realistic, such as a 4 km x 4 km site with < 50 images. I can also review your recipe and data for a smaller site. One may even make a final product in a piecewise manner (with some careful work).

[steo85it]

Thanks a lot, Oleg! I have been testing this yesterday and it seems to work (more or less) as intended. As you mentioned here and in the notes, the resulting transformation is not "exact" and, at least in my small test case, it tends to overshoot ... I'm trying to see if I can make it converge by iterating (it doesn't look like it). Else, yes, I'm already working on 10x10 km "tiles" to be later aligned (hopefully not too much, if this step works) and mosaic-ed. I'll let you know, thanks again!

[oleg-alexandrov]

Else, yes, I'm already working on 10x10 km "tiles" to be later aligned

That must be one heck of a site if the tiles are that big. If the tiles have overlap, say maybe 250 m - 500 m, or ideally a bit more, the pc_align tool with the option --initial-transform-from-hillshading (and maybe zero iterations) does something similar to image_align, finds interest point matches and creates a transform. That one can be used to shift SfS DEM tiles if the tiles don't agree to each other.

It is still recommended to also shift the cameras for that tile as well, and do another bundle-adjust for that tile with a DEM constraint to ground a bit more the shifted images to the new terrain. There's also the pc_align options --initial-ned-translation and --initial-rotation-angle which, with 0 iterations, can nudge a DEM by desired amount in local coordinates. These knobs are not a recommended approach, just when one has no choice really. It is also possible to use dem_mosaic to refine a small subtile and blend in the larger result if local fixes are needed.

[steo85it]

Thanks Oleg, sorry for the delay, I have been testing your implementation. Is there any reason why there would be a "factor 3 error/overshoot" in the conversion? If I correct for that factor, I find a very good match in the cameras (i.e., very close to the alignment I get on the projected rasters). I just tested this at a location for the moment, so it might be "a lucky factor", too: still thought I could ask if that rang a bell while checking somewhere else, too.

Else, yes, pretty large "site" (or rather "strip"). Thanks for the suggestions! The tiles overlap by 1km on each side, so I'm indeed pc_aligning them as a final alignment+validation before dem_mosaic. Only, for some reason, pc_align usually performs way worse than image_align for me (not sure about the detailed differences between the 2 algos).

It is still recommended to also shift the cameras for that tile as well, and do another bundle-adjust for that tile with a DEM constraint to ground a bit more the shifted images to the new terrain

To correct for the imperfections in the sfs dem resulting from the misaligned images in the first place, you mean? That sounds recommendable, yes, and somehow also what's making me "iterate forever" in some cases. ^^

[oleg-alexandrov]

Is there any reason why there would be a "factor 3 error/overshoot" in the conversion?

That is odd. There's multiplication by any factor as part of the calculation. Just a transform is fit to the data. Not sure.

Only, for some reason, pc_align usually performs way worse than image_align for me (not sure about the detailed differences between the 2 algos).

I know that the pc_align algorithm does not work well for small SfS clips, especially if those are made with images that may have been misaligned. Then the fine-level texture created by SfS may be out-of-sync with the large-scale features. This tool however should work well with two decent DEMs having mountains, crater rims, or other notable features. You can try doing a profile along some line of your input DEMs (say in QGIS) and see if they are similar to start with and have notable relief. Then one can try to use this tool.

It is still recommended to also shift the cameras for that tile as well, and do another bundle-adjust for that tile with a DEM constraint to ground a bit more the shifted images to the new terrain To correct for the imperfections in the sfs dem resulting from the misaligned images in the first place, you mean?

If you have a set of self-consistent cameras, and you get a nice self-consistent SfS terrain, but that terrain is misaligned to the true ground, and then you move those cameras using the same alignment transform to align better to the new ground, they will still be self-consistent, but no longer so well-tied to the ground itself. For example, they may need to be also brought down a bit.

That sounds recommendable, yes, and somehow also what's making me "iterate forever" in some cases. ^^

Alignment is tricky business. There must be alignment among the cameras, and alignment to ground. And the kind of scale you work at is guaranteed to be a big time sinc, on the order of a month, if not a lot more. That even before the tools start failing because of inadequacies of a particular dataset or site.

[steo85it]

I forgot to update here: never mind my previous comment, your implementation of the ecef-transform works great and has been super helpful and reliable for all my applications up to now (it just failed for a few cases with very large misalignments, but they are easy to identify and correct/remove)! Thanks again!

[oleg-alexandrov]

Glad to hear it worked. At some point we should make it possible to scale a CSM camera to make it lower-resolution, which would have made your job easier. Here in the meantime I redid (successfully) an SfS example with 814 full-res LRO NAC cameras close to the pole on a 14 km by 11 km site which was tricky to get right because of the many large images, large site, and very diverse illumination (the doc got a big revamp as well). Thanks for putting up with the tools.