Building PySCeS with the new versions of numpy, VStudio and gfortran

[bgoli]

If you try building PySCeS with a newer version of numpy/f2py, Visual Studio and the MinGW GCC supplied with Ananconda you may run into a "unknown flag" type error when compiling the fortran modules. This is due to the old version of GCC/gfortran supplied with Anaconda.

The following recipe has been tried and tested by Jannie Hofmeyr and @bgoli (note that steps 7-9 are probably not necessary if you have a full install of Visual Studio)

1. Download python 3.7.7 from www.python.org. Install python 3.7.7 (in c:\python37; choose add to path)

2. pip install numpy scipy matplotlib ipython jupyter pandas sympy nose (suggestion from scipy.org)

3. pip install pysces (Everything should works perfectly)

4. Download mingw64 from https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/threads-posix/seh/x86_64-8.1.0-release-posix-seh-rt_v6-rev0.7z/download

5. Unzip mingw64 to root (c:\mingw64)

6. Add c:\mingw64 to path (check that it occurs before any other gcc - in my case the gcc of strawberry.perl)

7. Download Microsoft Build Tools for Visual Studio 2019 from https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019

8. Install C++ build tools and Windows 10 SDK (instruction from https://wiki.python.org/moin/WindowsCompilers)

9. pull the latest pysces from https://github.com/PySCeS/pysces

10. python setup.py build --compiler=msvc --fcompiler=gfortran install bdist_wheel

11. pip install pysces-0.9.8-cp37-cp37m-win_amd64.whl

RESULT

In [1]: import pysces c:\python37\lib\site-packages\pysces\PyscesPlot2.py:749: MatplotlibDeprecationWarning: The 'warn' parameter of use() is deprecated since Matplotlib 3.1 and will be removed in 3.3. If any parameter follows 'warn', they should be pass as keyword, not positionally. matplotlib.use(backend, warn=False) Matplotlib backend set to: "TkAgg" Matplotlib interface loaded (pysces.plt.m) Continuation routines available NLEQ2 routines available You are using NumPy (1.18.2) with SciPy (1.4.1) RateChar is available Parallel scanner is available

PySCeS environment

---

pysces.model_dir = C:\Pysces\psc pysces.output_dir = C:\Pysces

---

• Welcome to PySCeS (0.9.8) - Python Simulator for Cellular Systems *
• http://pysces.sourceforge.net *
• Copyright(C) B.G. Olivier, J.M. Rohwer, J.-H.S. Hofmeyr, 2004-2020 *
• Triple-J Group for Molecular Cell Physiology *
• Stellenbosch University, ZA and VU University Amsterdam, NL *
• PySCeS is distributed under the PySCeS (BSD style) licence, see *
• LICENCE.txt (supplied with this release) for details *
• Please cite PySCeS with: doi:10.1093/bioinformatics/bti046 *

  ---

[jmrohwer]

Update on / additions to the above recipe:

• I have been using a similar approach myself for a while now and tested this on Python versions 3.6-3.8 (each downloaded and installed separately from www.python.org. (The PATH has to be adjusted by hand to activate different python versions on the command line if they are installed in parallel. I have written a series of scripts to do this).
• Step 2: For building only, numpy on its own is sufficient. The rest are not needed in a build environment. The working environment is of course a different story but ideally the two shouldn't be mixed.
• Step 3 is not necessary. The point is to build pysces, why pip install it?
• Step 10 can be simplified to:
  python setup.py bdist_wheel
  Numpy distutils automatically sorts this out and will use MinGW (gfortran) for compiling the Fortran and MS Visual C++ for compiling the C code.
  See also: https://pav.iki.fi/blog/2017-10-08/pywingfortran.html

Finally, I am building binary wheels for Windows with every new PySCeS release and uploading them to PYPI. There should thus be no need to compile your own, a simple pip install pysces should do the trick...

[bgoli]

Update Anaconda3 now uses clang and flang as compilers (+1). All you need to do is make sure -c conda-forge ... fortran-compiler wheel to your environment and you are good to go as far as fortran is concerned. See INSTALL.md for details. I need to test the portability of the binaries but this looks a better solution than messing around with mingw. Also we can add using Visual Studio Community Edition (currently 2019) for c/c++ support.

[jmrohwer]

Great, I tried this on a Windows VM, and could get Pysces to build. Have you gotten the Anaconda packages to build in the same way, i.e. without mingw? I tried by making the following changes in the recipe:

• add fortran-compiler as a build requirement
• change the script to: script: "{{ PYTHON }} setup.py build --compiler=clang --fcompiler=flang install --record=record.txt" OR script: "{{ PYTHON }} setup.py install --record=record.txt"

BUT it didn't work and in both cases the build bombed out complaining about it can't find gfortran. Have you tried this? Would be cool to get rid of the mingw dependency altogether.

[bgoli]

I would go the first route did you specify conda-forge as a required channel?

[jmrohwer]

Actually I made both changes in the recipe. The script has to be changed anyways, as it currently reads: script: "{{ PYTHON }} setup.py build --compiler=msvc --fcompiler=gfortran install --record=record.txt"

And I'm running miniforge, so conda-forge is actually my default channel. Still got the errors though. Will do some research.

[bgoli]

I haven't tried with recipebuild bdist_conda seems to work, so we could just script a whole bunch of build/test environments on Windows.

[jmrohwer]

Update: So I was able to build a conda package with bdist_conda as well as normal Python wheel with bdist_wheel.

BUT: the required libraries are not bundled with the binary packages. This means that the normal wheel installed into a plain Python environment (not Anaconda, but from python.org) with pip will complain about DLL not found when trying to import pitcon or nleq2.

Also, installing the binary Anaconda package into a clean environment gives the same DLL not found error. Only when you conda install fortran-compiler into that environment as well does it work. We cant expect all our users to install fortran-compiler into their own environments.

So, while this seems to be a great option for people wanting to do their own builds from source, it does not appear to a workable option for preparing binary packages for distribution, and we may have to stick to the scripts as in packaging\conda (which is all set up on my machine BTW so I don't mind doing the builds).

If you can get this going, please let me know, it would indeed be great to get rid of the gfortran dependency.

[bgoli]

I'm sure is a a better way of dealing with these fortran dependencies but for now I was assuming we would use your configuration for building packages :-)

This would be my recommended way of building your own version of PySCeS with Anaconda which is easier than any other method.

[jmrohwer]

I'm closing this now. The above instructions have been added to INSTALL.md for building your own version with the Anaconda compilers. Building portable packages is now also properly documented with README.md files in the packaging/conda and packaging/many_linux directories.