Closed bekozi closed 8 years ago
That is a good question! We need to make that clearer somehow...
Right now there is no clear way to declare them for all CIs. We add them separately in the .travis.yml
for OS X and the Dockerfile
for Linux. On Windows we must package them! (See the expat
and the udunits2
recipes for an example.)
I see! It would be nice to have platform pages where recommended packages could be listed for yum
/ apt-get
/ brew
, etc. Package-specific might be too much to manage, but a list of recommended Ubuntu packages (for example) seems reasonable. Could sections be added in the "Setting up the IOOS Python Environment for..." pages?
package-specific might be too much to manage
It is indeed but necessary for build reproducibility.
Could sections be added in the "Setting up the IOOS Python Environment for..."
We can certainly improve the docs, but ideally no system library is needed to use the packages (only to build them). I am trying to eliminate that problem when it crops up. (See #667 for example.)
We can certainly improve the docs, but ideally no system library is needed to use the packages (only to build them).
Ahhh, okay. Thanks for the clarification. So for ESMF
this installation test fails:
docker run continuumio/miniconda bash -c 'conda install -c ioos esmpy && python -c "import ESMF; m = ESMF.Manager(); print(m.pet_count)"'
Traceback (most recent call last):
File "/opt/conda/lib/python2.7/site-packages/ESMF/interface/loadESMF.py", line 111, in <module>
mode=ct.RTLD_GLOBAL)
File "/opt/conda/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libgfortran.so.3: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/conda/lib/python2.7/site-packages/ESMF/__init__.py", line 66, in <module>
from api.esmpymanager import *
File "/opt/conda/lib/python2.7/site-packages/ESMF/api/esmpymanager.py", line 10, in <module>
from ESMF.interface.cbindings import *
File "/opt/conda/lib/python2.7/site-packages/ESMF/interface/cbindings.py", line 12, in <module>
from ESMF.interface.loadESMF import _ESMF
File "/opt/conda/lib/python2.7/site-packages/ESMF/interface/loadESMF.py", line 114, in <module>
raise ValueError('The ESMF shared library did not load properly.')
ValueError: The ESMF shared library did not load properly.
This will pass because we are installing gfortran
(turns out for this test case build-essential
is not needed:
docker run continuumio/miniconda bash -c 'apt-get -y update && apt-get -y install gfortran && conda install -c ioos esmpy && python -c "import ESMF; m = ESMF.Manager(); print(m.pet_count)"'
From what you said, we would want to solve this upstream in the recipe - trying to build with conda's libgfortran
for example. Is this correct? In the past, we addressed this by providing our own "gcc-with-gfortran" package. This was before conda provided a Fortran compiler.
From what you said, we would want to solve this upstream in the recipe - trying to build with conda's
libgfortran
for example. Is this correct?
Yes, if possible, if is does not work...
In the past, we addressed this by providing our own "gcc-with-gfortran" package. This was before conda provided a Fortran compiler.
.. we will go with what you know to work :wink:
The goal is to use as much as we can from the default channel. That way we avoid putting an extra burden on the channel maintainer.
Ideally, if we need something that is"broken" in the default channel we should push a fix to the default channel rather than maintaining our own build.
There is a significant increase in channels out there, providing different recipes for the same package. I think that this is harmful to the scientific community and we should strive to avoid that kind of "recipe forking" as much as we can by pushing changes to community channels and/or the default channel.
@bekozi can you test if installing libgfortran
fixes that? If so we can just add is as a dependency rather than re-building it with conda's or a custom gcc
.
Installing libgfortran
doesn't fix the issue (tried that already). There will need to be some rebuilding and/or re-linking. Let me do some experimentation with the default channel compilers and see how it goes... Wish me luck. :-)
Installing libgfortran doesn't fix the issue (tried that already).
:disappointed:
Let me do some experimentation with the default channel compilers and see how it goes... Wish me luck. :-)
Finger crossed!
I've done some experimenting with Linux build set-ups. OSX and Windows may have their own set of challenges.
gcc
now provides gfortran
. Nice. At the very least, the default channel gcc
requires libc6-dev
on Ubuntu and glibc-devel
on CentOS6.gcc
/gfortran
installation is not expected on user machines, netcdf-fortran
and ESMF
need to be rebuilt with gcc
as a build/run dependency.mpich2
has Fortran compiler linking issues when attempting to build ESMF
(probably anything else as well). Building a new mpich2
using the default channel gcc
addresses the issue. mpi4py
may also need to be built.The draft changes/additions to build recipes can be viewed here: https://github.com/ioos/conda-recipes/compare/master...NESII:esmf-libgfortran.
There are two paths to addressing this issue from these tests. To be clear, the issue is users must have gfortran
installed on their machine before using IOOS's ESMF
.
gfortran
.gcc
requirement to at least netcdf-fortran
and ESMF
. Either build mpich2
with default gcc
or fix the default channel mpich2
.Option 1 is definitely easier but not as inclusive. Option 2 seems more durable in the long-run. It may be painful to implement. There may be cascading issues in other packages by introducing gcc
. Option ? could involve fancy post linking.
What do you think?
Great work @bekozi! It is sad that the default channel "build tools" are not ideal yet.
I am :+1: for option (1)
Me too! I'd recommend text for Linux like below. I'd send a PR, but I don't think that's possible for wikis... Happy Holidays, BTW! :christmas_tree:
Most IOOS conda packages have no prerequisite system libraries. Exceptions are listed in the table below.
Prerequisite | Installation Instructions | Required By |
---|---|---|
GFortran | Install GFortran | esmf , esmpy |
GFortran may be installed using your default Linux distribution's package manager:
$ [sudo] apt-get install gfortran
or
$ [sudo] yum install gcc-gfortran
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus ac erat sem. Integer volutpat nibh ipsum. Nam posuere tempor sem eu venenatis. Aenean vitae semper odio, id posuere diam. Quisque in nunc interdum turpis accumsan pellentesque. Nullam ac diam nec nulla vulputate semper.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed consectetur elit tellus, sed laoreet justo vulputate rhoncus. Fusce laoreet non massa at aliquet. Curabitur efficitur lacus vitae ex iaculis, nec luctus eros lobortis. Fusce bibendum lorem orci, vitae rutrum leo commodo consectetur.
Sounds great! Thanks Ben.
I will leave that to our wiki master @rsignell-usgs :stuck_out_tongue_winking_eye:
Happy Holidays!
@rsignell-usgs: Have you had a chance to look over these suggested wiki additions? https://github.com/ioos/conda-recipes/issues/676#issuecomment-167015494 Thanks!
@bekozi I added a few extra dependencies to the new packages on conda-forge and this problem should be fixed. If not let's open a new issue there.
What is the best practice for indicating required system libraries for a conda package?
For instance, when installing
ESMF
on Ubuntu,build-essential
andgfortran
are required (i.e.apt-get install build-essential gfortran
). Also,libexpat1
is required forudunits2
. I am thinkingexpat
should maybe be a requirement forudunits2
?Thanks!