External zlib, htslib and pyflow

[roelj]

First of all, thank you for creating this structural variant and indel caller!

Is it possible to add configuration options to use externally available zlib, htslib and pyflow, just like using an external CMake and Boost? Ideally, something along the lines of:

./configure --with-zlib=/directory/to/zlib --with-pyflow=/directory-to-pyflow --with-htslib=/directory-to-htslib

(going for this approach, it would be nice to also have --with-boost)

I tried to provide a patch, but I don't understand the way the build system has been set up now.

[ctsa]

Thanks, I think we would like to do something similar for zlib anyway so I expect we can cover that one relatively soon. What is the motivation for wanting to customize the others? For htslib in particular we have to dig a little bit below (what I gather to be) the official API level, so I'm not sure we can offer much flexibility on the library version in a robust way.

[roelj]

Thanks. I think zlib should not be a problem anyway, because its API and ABI are quite stable.

As for the others: We would like to add it to GNU Guix (a cross-distribution package manager for reproducible software deployment), and we're almost there.

We have separate packages for boost, zlib and htslib, so sharing those among programs results in a more efficient distribution, and less code we have to maintain. It's not uncommon to ship different versions of libraries like htslib when there's a good reason to do so (there's little you can do when upstream does not have a stable API..). And when we do so, we get a pretty good idea about the reasons why.

[ctsa]

Thanks for passing on the link, exciting to see this work on package integration.

I understand the concern about vendoring dependencies, but for packages like htslib and pyflow, it's the best approximation to having manta "just work" from source for a user that's solely focused on their SV calls.

We can meet you partway here -- even if the zlib fix isn't directly applicable for your case anymore I agree this should be more flexible. I'll look into what can be done for htslib -- we should probably bump up to 1.3 at least so that we're back on a stable release version. If there's anything about the build system that's outright blocking you then let us know so that we can prioritize.

[roelj]

Can't we default to bundling them (like it works right now), but also allow for specifying an external version (by passing extra configuration options)? With GNU Guix we can use the exact same htslib and pyflow as what would be downloaded.

So that would suite your users because the default "just works", and it encourages an even better solution than compiling it yourself: Addition to a package manager that in turn allows for one-command installs: guix package -i manta.

I will do another attempt to fit it in the build system in a way that it "just works" for regular users and yet give the flexibility that is needed to unbundle it for those who really want to do that.

[tillea]

BTW, I intend to package manta for Debian and I patched the cmake files to use Debian packaged libraries. I'm not yet sure whether this approach was successful since I spotted an issue with the demo which I intend to report right now. I'm just mentioning it here since I spotted this issue when checking whether my problem was reported yet.

[roelj]

@tillea Can you share the patches?

[tillea]

On Fri, May 19, 2017 at 05:17:37AM -0700, Roel Janssen wrote:

@tillea Can you share the patches? Sure. All patches I applied to the Debian packaging can be found here: https://anonscm.debian.org/cgit/debian-med/manta.git/tree/debian/patches

[roelj]

Thanks a lot!

[roelj]

@tillea Your patches are wonderful. I had to disable the unit tests to pass the build for GNU Guix: https://github.com/UMCUGenetics/guix-additions/blob/master/umcu/packages/manta.scm