Binary dependency from SpecialFunctions

[GregPlowman]

I note that the latest release of Quadmath has a hard dependency on SpecialFunctions which has a binary dependency.

Similar to https://github.com/sunoru/RandomNumbers.jl/issues/53, this makes it harder to deploy in some environments.

In my case, I'm using DoubleFloats which depends on Quadmath which breaks because of SpecialFunctions when running parallel simulations across a cluster.

Is there some way to conditionally load SpecialFunctions? Or some other way to avoid this binary dependency. I guess splitting SpecialFunctions into SpecialFunctionsBase (function headers) and SpecialFunctions (implementation) could also work?

(As an aside, I presume libquadmath doesn't present the same problem because it is a system, rather than package dependency? Is this correct?)

[simonbyrne]

In my case, I'm using DoubleFloats which depends on Quadmath which breaks because of SpecialFunctions when running parallel simulations across a cluster.

What is the error you're seeing? There is no reason why this shouldn't work.

I presume libquadmath doesn't present the same problem because it is a system, rather than package dependency?

It's actually because libquadmath is bundled with Julia, as it is part of the gfortran runtime.

[GregPlowman]

What is the error you're seeing? There is no reason why this shouldn't work.

Can post exact error message when back at work. But the problem is that within our corporate environment, packages are installed on a central server, and clients load code from this central server. This works fine for Julia-only code, but for binaries it can be a problem. (This is true for homogeneous networks, but especially if different architectures are involved).

Of course, in the "normal" case where packages are installed on each individual machine, things work fine, but binary dependencies do limit deployment options, especially for clusters in corporate environments where users might not have full control.

I guess I can work around this by maintaining our own copy of Quadmath without SpecialFunctions, but I thought this might affect other users as well.

In the issue linked above (sunoru/RandomNumbers.jl#53), @ChrisRackauckas explains a similar issue with DifferentialEquations, where "it can be difficult to install in settings without internet access because binaries are architecture-specific". Perhaps this might also affect such users if they want to use Quadmath with DiffEq? Chris do you have any input here?

It's actually because libquadmath is bundled with Julia, as it is part of the gfortran runtime.

Ah thanks, that explains why this is not an issue.

[ChrisRackauckas]

Not anything more specific. If you have a binary, you can't copy-paste that package folder to a computer behind a firewall if it has a different operating system. Just those normal edge cases where it's nice to be pure Julia. Not a huge deal, but something I would at least like to fix for DiffEq.

[RalphAS]

Quadmath does not really depend on SpecialFunctions, but supplements it with appropriate methods. This was previously handled as conditional by means of runtime imports using Requires.jl (a design chosen to avoid issues like this, then hypothetical). Unfortunately the package manager does not fully understand Requires, so the prior implementation led to misleading warning messages. The author of DoubleFloats was sufficiently distressed by such warnings to write #35.

[GregPlowman]

Quadmath does not really depend on SpecialFunctions, but supplements it with appropriate methods.

Yes, which is why I suggested above that SpecialFunctions could be separated out into abstract function headers and separate implementation. Then other packages could extend those abstract functions without importing and depending on the binary implementation. Could this work?

Unfortunately the package manager does not fully understand Requires, so the prior implementation led to misleading warning messages.

RandomNumbers appears to conditionally import a module using the new package manager. Is that a similar case to this?

[RalphAS]

I think your suggestion is a good one; it would also facilitate development of a seamless pure-Julia implementation for common types.

Thanks for pointing to RandomNumbers; it uses a more current approach to Requires which avoids the warnings. I will try to correct the usage here soon if someone else doesn't do it first.

[GregPlowman]

Thanks for the PR @RalphAS

cc @simonbyrne, @JeffreySarnoff

Not really sure where to post this, but I have been thinking a bit more about this, especially now that DoubleFloats has been upgraded to have SpecialFunctions as a direct, unconditional dependency.

Some random thoughts:

There has been much discussion about what goes in Base vs stdlib vs packages, and I don't really want to rehash all the different aspects here, except to note that a binary dependency in Base or stdlib ships with the Julia installation.

So could this issue be resolved by bringing SpecialFunctions (or it's binary) in from the wild?

Maybe SpecialFunctions could be in stdlib? Or at least the binary (libopenspecfun) could be shipped with Julia.

Is Float128 close to being considered a "first-class" type now? As "first-class" as say BigInt or BigFloat? (What is the reason those types are in Base?) If so, would it make sense to ensure all binary dependencies are bundled with Julia? (especially since libquadmath is already)

[JeffreySarnoff]

For DoubleFloats its about Quadmath directly and SpecialFunctions appear because some of those functions are supported by libquadmath for the Float128s. Double64s cannot use special function implementations that are designed for and specialized to Float64s.

[GregPlowman]

For DoubleFloats its about Quadmath directly and SpecialFunctions appear because some of those functions are supported by libquadmath for the Float128s. Double64s cannot use special function implementations that are designed for and specialized to Float64s.

Yes, I understand that, which is why I suggested above that SpecialFunctions could be separated out into abstract function headers and separate implementation. That way, DoubleFloats would only depend on say SpecialFunctionsBase and the libquadmath implementation for Float128, and not the binary libopenspecfun implementation for Float64. Perhaps that's a longer-term plan.

The current situation is that DoubleFloats depends on the binary dependency of SpecialFunctions, even though it doesn't use it.

Distributing SpecialFunctions &/or libopenspecfun with Julia (Base or stdlib) would be a solution to this issue in the interim.

[JeffreySarnoff]

I agree with your "longer-term" plan; and agree more vociferously if it be done sooner rather than "longer".

[GregPlowman]

Yes sooner is better.

Interestingly, implementing the “longer-term” plan alone would result in the unusual situation where the SpecialFunctions implementation for Float128 is bundled with Julia, while the implementation for Float64 is in an external package.

So this issue would be solved for DoubleFloats & Float128, but would still exist for Float64.

I still think there’s a case for SpecialFunctions to be distributed with Julia, perhaps as a stdlib.

[JeffreySarnoff]

As Special Functions is an umbrella term, and there are domain-specific field favorites, it would be very nice to provide the communtiy with multi-practical availability and appicability. One recent approach to the organizational part of is fungrim. I think there another layer to the cake, there ought exist a picker-upper and auto-provisioner for every little thing. [only half kidding] That LinearAlgebra is a stdlib may help this get there.

[JeffreySarnoff]

My understanding of Float128 readiness is there remains something to resolve: the not-well-understood of-what-happens on Win after a few fma(x::Float128, y::Float128, z::Float128).

[simonbyrne]

I have tagged a new release with #38 included, so that should help address your immediate problems.

The medium term plan is to make binary dependencies available via the package resolver (https://github.com/JuliaLang/Pkg.jl/issues/841).

Maybe SpecialFunctions could be in stdlib?

That's not going to happen: since stdlibs need to be backward compatible, it is exceptionally hard to change or improve them (e.g. the LibGit2 and SparseArrays package are a frequent source of complaints in this regard). This will hopefully change once we have stdlib versioning, but then they would be "just" packages.

Or at least the binary (libopenspecfun) could be shipped with Julia.

There is no appetite for bundling code with Julia that isn't required to run Julia itself: since it would be untested, it is a recipe for unintentional breakage. Also, why should libopenspecfun be special in this regard?

Interestingly, implementing the “longer-term” plan alone would result in the unusual situation where the SpecialFunctions implementation for Float128 is bundled with Julia, while the implementation for Float64 is in an external package.

This is simply an accident of having libquadmath unintentionally bundled. There is no guarantee it will stay that way, and it may well go away in future releases, in which case this package will have to get its own binary dependency. Note that only a handful of special functions are provided by libquadmath.

For your specific circumstances, you may want to look into JuliaTeam (disclaimer: this is a product from my former employer, in which I have a very minor equity stake).

[GregPlowman]

OK fair enough.

I have tagged a new release with #38 included, so that should help address your immediate problems.

Yes, thank you.

Unfortunately, the latest version of DoubleFloats now has a direct, unconditional dependency on SpecialFunctions. @JeffreySarnoff, is there any chance a similar conditional dependency could be implemented for DoubleFloats?

[JeffreySarnoff]

Sure -- no problem. What? I can follow directions.

I removed SpecialFunctions from the [deps] section, it was in the [extras] section already. I removed the using Special functions line from the top level file.

I created a file that begins

import .SpecialFunctions
import .SpecialFunctions: erf, erfc, besselj0, besselj1, bessely0, bessely1,
    besselj, bessely, gamma, lgamma, loggamma

then includes all files that use names of functions in SpecialFunctions, and now that one is included into the top level file and the files it gathers are no longer are directly included

There may be touch-ups needed before these changes make happen what you expect to happen.

[GregPlowman]

OK this is terrific. Thanks!