Build ASP using modern tools

[oleg-alexandrov]

Nowadays there is infrastructure to find dependencies, build, and test software in the cloud, for Linux and Mac (heck, even for Windows for almost all ASP dependencies). A nifty tool is conda-smithy that can automate most of this and connect to GitHub and TravisCI, etc, for fetching latest code and do the build and testing.

Almost all of our dependencies apparently exist already on conda-forge, including all ISIS dependencies, then also bzip2, pbzip2, imagemagick, chrpath, openjpeg, geos, xz, gdal, hdf5, parallel, cspice, laszip, gflags, glog, and ceres. Just a handful will need come custom recipes. It is nice to see how the software ecosystem is evolving and we should take advantage of that.

[dshean]

An official conda-forge ASP/ISIS install would be fantastic. Bonus for ASP Windows build, which would definitely expand the user base.

[jomey]

Started an initial conversation with @ocefpaf and how he could see a path for ASP to be available as a conda package. Hopefully there will be not too many custom recipes.

[oleg-alexandrov]

Any thoughts and ideas are appreciated. I think the core stereo and bundle adjustment tool would not need any custom packages. Those are needed for pc_align and camera_solve, which could be dealt with later.

One issue I ran into recently was that the imagemagick on conda-forge could not gets its dependencies reconciled with the system I inherited from ISIS. But this one is again needed just for peripheral tools.

The key check would be if the GDAL in conda-forge can play nice with the rest of our conda packages and if ASP would build with that version (we now are at GDAL 2.0, I don't know what changed in the latest GDAL, which is 3.0). If there is no big pain here, other things are manageable.

In the meantime I learned that conda-build is good at not just building but packaging as well, so dealing with rpaths and other portability adjustments. That is also some logic we have a homegrown tool for and I'd be happy to have it offloaded..

[jlaura]

Some thoughts:

• This is awesome to see!
• The major change I saw in GDAL3.x was the move to proj5 or proj6. This caused breakages in brittle tests that were looking to parse out some gdalinfo type metadata from project strings. The format of those strings has changed in proj > 4...
• We are going to be working to get ISIS installable alongside GDAL in the coming FY. That should help with an ISIS dependency in a larger scientific computing stack. If you are pulling ISIS from our conda channel, this is going to be a short term issue since gdal/isis is not possible at the moment.
• If ISIS needs to be in the same env as ASP (long term it totally should be I think), we might also have a snag at QT. We have some extra dependencies that are not in conda-forge and are maintaining a snowflake there. In FY20, this is something we are going to try and address.
• For your binaries, why aren't you shipping all of those in a single package? (I suspect I am misunderstanding the above comment.) For example, we ship all of the ISIS binary executables as a single unit. The downside to this is that a conda activate my_isis_env has either a very clean PATH or a very polluted PATH. It all depends on your perspective as the <my_isis_env>/bin directory is prepended to the PATH.
• Would a FindISIS.cmake script be useful on your side? I believe we can make one of these with basically zero effort because some kind CMake dev. added an autogenerate feature 5+ minor versions back.
• Epically in the weeds here. You are probably going to want to wrap your enable_testing call in the CMakeLists root file in a variable and disable it on the Conda-forge side of things. No reason to test there when you have standard CI on the dev side and are just using conda-forge to package/release.

[oleg-alexandrov]

Thanks a lot. Good to know about GDAL.

Yes, hoping to have ASP and ISIS in the same env, it just makes sense.

I guess we will ship our stuff as a single package. I meant to say, we've been doing our homegrown packaging, and that is extra code to maintain.

We already have cmake logic to find ISIS. Not something terribly standard but it works.

Good to know about having to change the testing framework.

All this will add up to some work, especially since some of these things I will have to learn about and hack around. But will make longer term maintenance much easier.

[jomey]

Hey Oleg, Awesome to see the added support to install asp via conda. 🎉 Are there more plans on using conda or is this ticket considered closed?

[oleg-alexandrov]

Good point. Conda support has been implemented.

There are probably things to do, particularly, making ASP and GDAL live in the same world, once that is figured by ISIS, but that is for later.