Make pip installable, migration to pyscaffold, f2py extensions

davidovitch commented 8 years ago

@pbrod @ocefpaf I am trying to get a new setup.py file up and running with pyscaffold, and the re-organized file structure as we discussed in PR #10. This all works fine until I need to compile the extensions with f2py. pyscaffold works fine with setuptools, but does not seem to be supported by numpy.distutils which is required in order to built the f2py extensions.

So what I have now is either:

setuptools + pyscaffold - f2py extensions
numpy.distutils + f2py extensions - pyscaffold (=manual package configuration)

I have looked at scipy and pandasfor inspiration, but they all go around without pyscaffold. Does anyone of you have any experience with this or useful tips?

ocefpaf commented 8 years ago

I never used pyscaffold, but I think it is just an organizer/cookie cutter, right? If so, use it once to get things organized and then just drop it.

I guess that the best way to go with Fortran extensions is numpy.distutils.

davidovitch commented 8 years ago

pyscaffold seems to be doing a fair amount of stuff when using setuptools.setup(use_pyscaffold=True). It determines the version based on the git repo among other things, but I haven't quite figured out what exactly.

davidovitch commented 8 years ago

I think I found the solution! In an old mailing list entry it was stated that when setuptools.setup was imported before numpy.distutils, like so:

from setuptools import setup
from numpy.distutils.core import setup, Extension

the latter would use setuptools. In doing so, you can use setuptools.setup in combination with f2py extensions!

ocefpaf commented 8 years ago

How is that different from using only from numpy.distutils.core import setup, Extension?

davidovitch commented 8 years ago

in that case you will not have the ability to let pyscaffold (with the use_pyscaffold=True option passed on to setup()) handle stuff (using the setup.cfg configuration file, distribution package creation, version tag creations, documentation building commands, etc)

davidovitch commented 8 years ago

I have pushed the work I have done so far to the pipinstall branch. Note that not all extensions are build correctly yet. This is very much a work in progress (WIP). I hope that by next weekend things are starting to fall into places. I do not have a Windows environment setup for testing yet, so I can only cover the Linux tests for the time being. @pbrod please let me know if the changes I have made so far are not in line with the direction you had in mind.

pbrod commented 8 years ago

@davidovitch I think this is a very good start. If possible I would like to keep pyscaffold because the setup script get much tidier than with only using distutils. In addition as you mentioned it will automatically keep track of the versions in a very clean way. Automatic versioning will simplify the release cycle.

davidovitch commented 8 years ago

@pbrod, I see you've also tried to compile the extensions with one interface file and several module/object dependencies. I am still clueless as to how to solve this, so I've posted a question on the NumPy mailing list here.

I've also bumped into complicated_build, but I am not sure I understand exactly what it does or how it could help.

pbrod commented 8 years ago

I discovered that the fortran files had tabulators' (\t) and they are a source for compilation errors, so I replaced those with spaces. What I find strange is that I am not able to even compile the test_fimod.f, test_rind71mod.f or test_rindmod.f on my windows 7 machine. I suspect there is something wrong with my fortran installation. When I compile mvnprd, cov2mod and rindmod I get the following error:

Found executable C:\Users\pab\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\link.exe mingw32.lib(lib32_libmingw32_a-mingw_helpers.o) : error LNK2005: decode_pointer already defined in MSVCRT.lib(MSVCR90.dll) mingw32.lib(lib32_libmingw32_a-mingw_helpers.o) : error LNK2005: encode_pointer already defined in MSVCRT.lib(MSVCR90.dll)

However, I am able to compile mvn.pyf and mvndst.f into mvn.pyd without errors.

I will try to change fortran installation and see if that helps.

pbrod commented 8 years ago

A possible solution is to try Cython and Fortran 2003 ISO C bindings. It is the only portable way to interop between Fortran and C (including CPython) according to this

davidovitch commented 8 years ago

Thanks to this reply on the Numpy mailing list we now have an example from Scipy here that does exactly what we are after. I've updated setup.py (see wafo-project/pywafo@05b46497c4172704bdebdd69d0eb92a1072894e5) which now compiles all extensions. Some issues remain:

All *.mod files end up in my wafo root folder instead of the temporary build directories, not sure what is going on here. The installed package directory looks nice and clean though.
I have not run the tests or looked at any results yet
wafo.mvnprdmod gives me the following import error:

ImportError: python2.7/site-packages/wafo/mvnprdmod.so: undefined symbol: __epsmod_MOD_eps

davidovitch commented 8 years ago

Test results on my machine (Linux 64-bit):

>>> import wafo
fig import only supported on Windows
fig import only supported on Windows
wafo/gaussian.py:19: UserWarning: mvnprdmod not found. Check its compilation.
  warnings.warn('mvnprdmod not found. Check its compilation.')

>>> wafo.test(label='full')
# clip output
----------------------------------------------------------------------
Ran 3095 tests in 290.862s

FAILED (KNOWNFAIL=17, SKIP=7, errors=27, failures=8)

At least two failures are due to the mvnprdmod extension having an undefined symbol.

If the current branch also builds on Windows I am +1 for merging into master and continue the development over there. I propose to start tackling the failed tests from the master branch. Note that I can also successfully create a wheel package (that includes the compiled extensions) as follows:

python setup.py bdist_wheel -d dist

davidovitch commented 8 years ago

@pbrod I would like to propose to merge this into the master, and continue to fix issues from there. What are you thoughts on this? Do you prefer I fix some of the issues first, or can I just go ahead with it?

pbrod commented 8 years ago

It looks good to me, so just go ahead and merge into master.

(For me the fortran builds crashes on both branches).

The most important issue is to get the build chain for the fortran modules working on travis. Then we will have much more confidence on new stuff coming from pullrequests.

I have put much effort into this but have not succeded. I will try to pick this up again this weekend. (No promise)

On 14 January 2016 at 16:58, David Verelst notifications@github.com wrote:

@pbrod https://github.com/pbrod I would like to propose to merge this into the master, and continue to fix issues from there. What are you thoughts on this? Do you prefer I fix some of the issues first, or can I just go ahead with it?

— Reply to this email directly or view it on GitHub https://github.com/wafo-project/pywafo/issues/14#issuecomment-171681952.

davidovitch commented 8 years ago

I've merged the pipinstall branch, but it seems I have introduced some new issues as a lot more test cases fail now... At this stage I don't know what is going on exactly. To be continued...

davidovitch commented 8 years ago

I've had an issue with my virtual environment. When using a miniconda environment as follows:

conda create -n py27 python=2.7
source activate py27
conda install scipy pandas matplotlib cython xlrd pytables sphinx nose

Build install wafo... ...and running the tests yields:

>>> import wafo
fig import only supported on Windows
python2.7/site-packages/wafo/gaussian.py:19: UserWarning: mvnprdmod not found. Check its compilation.
  warnings.warn('mvnprdmod not found. Check its compilation.')
fig import only supported on Windows
>>> wafo.test(label='full')

Which returns in the end:

----------------------------------------------------------------------
Ran 3095 tests in 274.857s

FAILED (KNOWNFAIL=17, SKIP=7, errors=10, failures=9)

So that is consistent with the situation as before merging the pip-install branch into master. There's still more to be done, especially fixing the build on Windows. I have access to a Windows machine so I could also give it a try at some point...

@pbrod when reading up on the build situation on Windows, I learned that is somewhat messy based on the following resources:

Using Microsoft Visual C with Python
Compiling C extension modules on Windows. The Cython wiki re-iterates what is said elsewhere: MinGW 64-bit is not in a usable or stable state. Otherwise 32-bit is ok (if your Python version is also compiled with that)
Building extensions for Python 3.5 (on Windows)

ocefpaf commented 8 years ago

@davidovitch Just to provide a few more data points. I tried to build the pipinstall branch using conda and our CIs. OS X and Linux are failing with ImportError: cannot import name futil

I guess that is related with the fact that the futil Fortran extension is not present or not compiled. (I believe that is no longer present in the modern SciPy code BTW.)

On Windows, using mingw, I get: https://ci.appveyor.com/project/comtbot/conda-recipes/build/1.0.1861/job/2urxc9wq6j4y3x41

I will test with mingwpy and let you know.

wafo-project / pywafo

Make pip installable, migration to pyscaffold, f2py extensions #14