Dependencies on libraries

[ChrisBarker-NOAA]

What should the policies be about dependencies on libs?

Key question: what system libs can be depended on.

Secondary question -- what should be done with libs needed:

• statically link ?
• ship dll, so, etc with package
• provide separate packge for lib (dll, so, etc.)

@JanSchulz wrote (in issue #16 ) (with my comments):

IMO: Linux: xserver but nothing else?

Plus the core libc of course -- essentially what Anaconda is built on .

Windows: no extra installed packages but the MS compilers, e.g. repackage r-tools and libswhich are > needed (tk, ...). No special libpath locations (old mpl recipe in conda-recipes),

+1

no extra installations of dependencies (e.g. python-igraph needs igraph (c lib) as dependency).

I don't get this -- did python-igraph require that you separately install igraph?

This would mean that we have to put c libs into the same conda channel as the python libs.

Yes -- that is definitely what should be done. If you are packaging a python wrapper around alib, the lib itself should be provided as a conda package.

Also interesting: what to compile in as a static dependency: I have the feeling that on windows pretty much everything is included in the final package and not used via a dll file...

yeah, it's looking that way -- I assumed that part of the POINT of conda was to be abel to provide libs that various packages could use. But I've been wrestling with this -- having to hack in a change to the PATH environment variable at run time -- ugly! But conda may have fixed this -- I think the LIbrary dir is now added to path in miniconda/anaconda -- but I haven't tested yet.

But if it can work, I think we should prefer that libs are installed as dlls by a separate package -- that way they can be used by multiple other packages. But statically link if it really is a only for this package lib.

[jankatins]

I don't get this -- did python-igraph require that you separately install igraph?

Yes: at least you have to compile igraph before you can compile python-igraph.

[jankatins]

But if it can work, I think we should prefer that libs are installed as dlls by a separate package -- that way they can be used by multiple other packages. But statically link if it really is a only for this package lib.

+1

That would mean that one could install python-igraph without igraph. Is it possible to specify two files as "source"? On the other hand: there is both python-igraph and r-igraph, so in this case it even makes sense to install igraph as a lib on it's own. But there are probably a lot of python packages around c libs which are only used as python package...

[ChrisBarker-NOAA]

Is it possible to specify two files as "source"?

Good question, I don't think so :-( Though you could do someting really hackish like calling curl inside your build.sh (not sure how on Winows...)

what I"ve done it either:

make the lib a separate package or put a copy of the source of the wrapped lib in my repo -- kind of ugly, but it does assure that the wrappers are all in sync with the source lib...

See: https://github.com/NOAA-ORR-ERD/orr-conda-recipes/tree/master/libgd and https://github.com/NOAA-ORR-ERD/orr-conda-recipes/tree/master/py_gd

for an example of keeping them separate.

Though I'm still struggling with getting it all to link right!

[pelson]

what I"ve done it either: make the lib a separate package or put a copy of the source of the wrapped lib in my repo -- kind of ugly, but it does assure that the wrappers are all in sync with the source lib...

We don't want to get draconian on this, but there should be clear guidelines, that the best approach is to separate the concerns into separate packages.

[jjhelmus]

To focus on what libraries dependencies should or should not be packaged on Linux I'll submit the following proposal.

For building Python wheels that work on a number of Linux platforms the Python community has codified in PEP0513 a list of libraries which define the "manylinux1" platform. I would suggest that conda-forge Linux conda packages target this same platform. The libraries include are:

libpanelw.so.5
libncursesw.so.5
libgcc_s.so.1
libstdc++.so.6
libm.so.6
libdl.so.2
librt.so.1
libcrypt.so.1
libc.so.6
libnsl.so.1
libutil.so.1
libpthread.so.0
libX11.so.6
libXext.so.6
libXrender.so.1
libICE.so.6
libSM.so.6
libGL.so.1
libgobject-2.0.so.0
libgthread-2.0.so.0
libglib-2.0.so.0

Which on RPM based systems these are provided by the packages:ncurses libgcc libstdc++ glibc libXext libXrender libICE libSM mesa-libGL glib2

I put forward that we yum install these packages on the Docker file we use for Linux builds.

PEP0513 also specifies maximum versions of glibc and glibcxx for compatibility. These correspond to the library versions provided in CentOS 5.11. These are older than the versions of the libraries we are currently using in our CentOS 6 based Docker image. Would it make sense to use a CentOS 5 based Docker image, perhaps using the manylinux1 or holy-build-box image?

[ChrisBarker-NOAA]

note that PEP 0513 was inspired, and pretty much cribbed, from what Anaconda (and Enthought Canopy) are already doing. Nice to have it written down though.

I presume we can also count on anything in the default channel....

[pelson]

I think basing our docker image off the manylinux image is a very sensible idea. I'm not super keen to go beyond those requirements, and we may find that in time we want to pare even some of those dependencies.

@jjhelmus - how is your docker? Interested in maintaining the Dockerfile which extends the manylinux docker images? I'm trying to figure out the best place for them to live in conda-forge - I guess for now they could live in conda-smithy until we have a better place.

(P.S. https://github.com/pelson/Obvious-CI/blob/master/obvious-ci.docker/linux64_obvci/Dockerfile & https://github.com/ioos/conda-recipes/blob/master/Dockerfile)

[jjhelmus]

My docker is a bit weak (I am a big vagrant users) but I have been looking for a reason to learn more. I'd be happy to maintain a manylinux like Docker image for conda-forge.

I've been playing with the manylinux Docker image this week and it really a great tool. I'll try to build a version similar for the project and submit a PR against conda-smithy.

[pelson]

Cool. You could also fudge it in a feedstock to prove it is working, and then apply any changes back to conda-smithy.

[jjhelmus]

Submitted the conda-forge/staged-recipes#97 PR to see if the new image works to build the example recipe.