Packaging libpll-2 for Debian Med

[shayandoust]

Hello,

Just a question about libpll and libpll-2. The Debian-Med packaging team has already packaged libpll1, however due to some softwares needing libpll-2 as a dependency instead (such as Raxml-NG), we could consider basing the source-code of libpll to libpll-2.

As libpll-2 is a fork of libpll, please could you confirm whether softwares written using libpll as a dependency can also use libpll-2? Are the functions preserved and do they have the same intended output?

Kind regards, Shayan Doust

[computations]

Hello Shayan,

libpll-2 is intended to be a super set of lipll. Libpll-2 came about as a fork to implement some fairly large optimizations that were deemed "too different" than the original library. So, a fork was made. Realistically, libpll-2 has replaced libpll, so it should be find to package libpll2 instead.

One thing to note before you end up doing a bunch of work: most of the products that use libpll-2 build and link it statically. I know this is true for my software, and I think that it is the default for raxml-ng. @amkozlov can speak further on this, but I personally have never linked with libpll-2 dynamically. Don't let this stop you from packaging libpll-2 though, just be aware of it.

Best Ben

[amkozlov]

Dear Shayan,

first of all, thanks for your efforts packaging our software, and sorry for the version confusion that we created! :(

As you can see from the commit history, libpll-2 evolved quite a bit since branching from libpll, so it's hard to guarantee that all software based on libpll will work seamlessly with libpll-2 (at least the might be "trivial" issues like unit tests failing due to minor numeric deviations in results, or an API function that got an extra "optional" parameter). But then, how many packages are using libpll at the moment?

Since you mentioned raxml-ng: I plan to release raxml-ng 1.0 in July, and this would be a good opportunity to introduce clean versioning for libpll-2 and pll-modules. We were a bit unsure about the future of both projects previously, but now it's clear that the codes are not being maintained/developed outside from our lab. So we shall have an internal discussion with everyone involved and implement whatever decision we come up with.

So my suggestion would be to wait with packaging just a little bit more.

Best, Alexey

[computations]

I decided to diff the last commit of libpll and the current head of libpll-2 for pll.h to see what the API difference was. Most of the changes are minor but here is a list

• Many structs and functions relating to site repeats.
• The functions pll_utree_query_tipnodes and pll_utree_query_innernodes have been removed.
• pll_utree_every and pll_utree_every_const have had their arguments changed. Previously it had been a call back function that took 1 node, and now it is a callback function that takes 2 nodes.
• const unsigned int* -> const pll_state_t* for arguments labeled tipmap.
• pll_core_update_sumtable_ti_4x4 has lost its tipmap argument.
• pll_core_likelihood_derivatives gained 2 additional arguments: unsigned int parent_ids and unsigned int child_ids.
• A utility function to parse phylip files has been added
• RNG functions have been added.
• Error codes for fasta and phylip files have gone from 1xx to 2xx.
• Some new fields in pll_partition_t: nodes, asc_additional_sites and repeats.
• Some new fields in pll_utree_t: binary and vroot.

I am not sure if any of those changes constitute a breaking change, but since the list was small enough I figured it would be worthwhile to enumerate the changes.

[shayandoust]

Hello,

First of all, I'd like to thank you both for warm praise of packaging your softwares and your prompt responses.

Libpll-2 came about as a fork to implement some fairly large optimizations that were deemed "too different" than the original library

Ahh, alright. Thanks for the explanation.

Don't let this stop you from packaging libpll-2 though, just be aware of it.

If there are any issues or concerns during packaging, I'll certainly raise them as a new issue here.

As you can see from the commit history, libpll-2 evolved quite a bit since branching from libpll, so it's hard to guarantee that all software based on libpll will work seamlessly with libpll-2 (at least the might be "trivial" issues like unit tests failing due to minor numeric deviations in results, or an API function that got an extra "optional" parameter)

In response to this and the list of differences @computations has kindly made, I think the best course of action to prevent any sort of "breakage" would be for us to simply package libpll-2 separately. This will ensure simultaneous versions that downstream and users can rely on (should their softwares link or take precedence with a version over the other).

But then, how many packages are using libpll at the moment?

This is quite a difficult question. Our simple best estimates is looking at popcon1 (though this is more of the "worst case" as not everyone opts to share data for popcon) or through sources.debian.org and finding where "libpll" has been referenced as a (build) dependency for all libraries and softwares.

As for raxml-ng, thanks for the insight. As you suggested, I might just hold back on the packaging until raxml-ng (1.0) is released, although raxml-ng is listed on our covid-19 packages that need work2, so there is a bit of urgency with packaging.

Thanks a lot for both of your efforts!

Shayan Doust

[shayandoust]

Hello computations et. al.,

Just a something that I've stumbled across. I've realised that libpll-2 generates libraries under the same name as libpll (as well as installing headers under /usr/include/libpll too). This will introduce conflict if a user decides to install both versions of libpll and libpll-2.

Is there a possibility of preventing this from happening within CMakeLists? Preferably, renaming to something like "libpll2" as opposed to "libpll" (which would mean libpll2-*.so files and headers installed under /usr/include/libpll2)? If it would make things easier with regards to not having a drastic upstream change, maybe a variable inside CMakeLists to allow for this to happen?

Kind regards, Shayan Doust

[computations]

We can (and should) change the name. The big hold up on our end is that we would want to update all of our products to the use the new name. It mostly just ends up being a time problem. Maybe this isn't as big of a deal as I am thinking, because the speed of development on this lib is pretty slow, so we actually have a large amount of time before we really need to change the name in dependent products. Nonetheless, it is a time cost.

If it would make things easier with regards to not having a drastic upstream change, maybe a variable inside CMakeLists to allow for this to happen?

Maybe I am misunderstanding this, but I think that having something that overloads include paths from libpll2 to libpll in the build process is just asking for bugs. My preference is to just rip the band-aid off and change the name wholesale, including all of our products.

In terms of time frame @shayandoust, what were you expecting (on the order of dats vs weeks vs months)?

[shayandoust]

Hello computations,

We can (and should) change the name. The big hold up on our end is that we would want to update all of our products to the use the new name. It mostly just ends up being a time problem. Maybe this isn't as big of a deal as I am thinking, because the speed of development on this lib is pretty slow, so we actually have a large amount of time before we really need to change the name in dependent products. Nonetheless, it is a time cost.

That does make sense, with a name change, you eradicate any possible conflicts further down the line when more softwares depend on libpll-2.

My preference is to just rip the band-aid off and change the name wholesale, including all of our products.

This would ideally be the best method.

In terms of time frame @shayandoust, what were you expecting (on the order of dats vs weeks vs months)?

There's no pressure of a timeframe, although of course the kind occasional upstream nudge to get an idea of progress :). Therefore, do carry out a name change whenever you find it feasible to do so. Although, the quicker we package this software and make it available in the repository, the more softwares we can support that depend on libpll-2 (for instance, raxml-ng which depends on pll-modules, which depends on libpll-2) and the less of a "blocked" state packages are.

Kind regards, Shayan Doust

[tillea]

Hi, I intend to continue Shayan's work. I confirm I consider it unfortunate to keep the same version as the old libpll as well as the same SOVERSION (which is set to 0). The latter means libpll-2 is a plugin replacement for libpll with full ABI compatibility. Can you confirm that this is the case? Kind regards, Andreas.

[xflouris]

Hi @tillea , the two libraries have diverged substantially. The cleanest solution would be to have libpll2 as a separate package, in the same way for example beast-mcmc and beast2-mcmc are maintained in Debian. The API differs, and ABI is also incompatible. There is some software using libpll but most new programs use libpll-2. Best regards, Tomas

[tillea]

Hi, Am Wed, Oct 20, 2021 at 01:25:37AM -0700 schrieb Tomas Flouri:

The API differs, and ABI is also incompatible. There is some software using libpll but most new programs use libpll-2. I absolutely agree - but the build system creates identical file names libpll0.{so,a} and you keep the very same version than libpll. I wonder whether you can change this and tag a new release. Kind regards, Andreas.

[amkozlov]

Hi @tillea,

many thanks for working on this and sorry for the confusion with libpll/libpll-2. We are now aiming to finally resolve it by replacing libpll-2 with a new library called coraxlib:

https://codeberg.org/Exelixis-Lab/coraxlib

We expect to switch our main tools (raxml-ng, generax etc.) to this library in the course of next year.

So from our perspective, it would make sense to wait until then and package coraxlib instead of libpll-2.

Best, Alexey

[tillea]

Hi Alexey,

Am Mon, Oct 25, 2021 at 07:13:14AM -0700 schrieb Alexey Kozlov:

many thanks for working on this and sorry for the confusion with libpll/libpll-2. We are now aiming to finally resolve it by replacing libpll-2 with a new library called coraxlib:

https://codeberg.org/Exelixis-Lab/coraxlib

Thanks a lot for the clarification.

We expect to switch our main tools (raxml-ng, generax etc.) to this library in the course of next year.

So from our perspective, it would make sense to wait until then and package coraxlib instead of libpll-2.

I admit I'm personally not under pressure. However, we have some package depending from raxml-ng which we want to finalise. So it would be great if you might manage until end of this year.

Kind regards, Andreas.

[computations]

We will try to get this done sooner rather than later, but barring that an intermediate solution would be to build raxml-ng statically. There is already an option to do this in the raxml-ng cmake: STATIC_BUILD. Maybe this is enough to hold us over until coraxlib is ready for shipment

[tillea]

Am Tue, Oct 26, 2021 at 12:44:18AM -0700 schrieb computations:

We will try to get this done sooner rather than later, but barring that an intermediate solution would be to build raxml-ng statically. There is already an option to do this in the raxml-ng cmake: STATIC_BUILD. Maybe this is enough to hold us over until coraxlib is ready for shipment Unfortunately that's no option for Debian packaging since its Debian policy to package separate projects (like coraxlib and raxml-ng) separately and do dynamic linking. I do not want to put real pressure onto you. I simply assumed that if you considered libpll-2 ready for shipment I naively assumed that renaming it would not be a complex step. But you will know better what means need to be done in addition. Kind regards, Andreas.

[tillea]

Any news about a release of this library?