Help installing on Win64 with MSYS2 (mingw64 compilers)

[kovalp]

Describe the issue

I am trying to get sisl installed on Windows, in plain Python installation (i.e. not Anaconda). My best take is to install MSYS2, install gcc and gfortran packages there and then pip gets farther as seen in the output below

koval@DESKTOP-B5CLTB9 MINGW64 ~/programs
$ pip install sisl
Collecting sisl
  Downloading sisl-0.10.0.tar.gz (3.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Requirement already satisfied: numpy>=1.13 in c:\users\koval\appdata\local\programs\python\python37\lib\site-packages (from sisl) (1.18.1)
Requirement already satisfied: pyparsing>=1.5.7 in c:\users\koval\appdata\local\programs\python\python37\lib\site-packages (from sisl) (2.4.6)
Collecting netCDF4
  Downloading netCDF4-1.5.4.tar.gz (793 kB)
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\koval\appdata\local\programs\python\python37\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\koval\\AppData\\Local\\Temp\\pip-install-v801fhjq\\netCDF4\\setup.py'"'"'; __file__='"'"'C:\\Users\\koval\\AppData\\Local\\Temp\\pip-install-v801fhjq\\netCDF4\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\koval\AppData\Local\Temp\pip-pip-egg-info-tyxwvna8'
         cwd: C:\Users\koval\AppData\Local\Temp\pip-install-v801fhjq\netCDF4\
    Complete output (16 lines):
    reading from setup.cfg...

        HDF5_DIR environment variable not set, checking some standard locations ..
    checking C:\Users\koval ...
    checking /usr/local ...
    checking /sw ...
    checking /opt ...
    checking /opt/local ...
    checking /usr ...
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\koval\AppData\Local\Temp\pip-install-v801fhjq\netCDF4\setup.py", line 397, in <module>
        _populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs)
      File "C:\Users\koval\AppData\Local\Temp\pip-install-v801fhjq\netCDF4\setup.py", line 345, in _populate_hdf5_info
        raise ValueError('did not find HDF5 headers')
    ValueError: did not find HDF5 headers
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Version details Run the below code and add to issue (if an issue is relevant for the issue):

import sys
print(sys.version)
import sisl
print(sisl.__version__)
print(sisl.__git_revision__)

$ ipython Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] Type 'copyright', 'credits' or 'license' for more information IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys ...: print(sys.version) 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)]

import sisl is not working yet.

[kovalp]

If I am installing using setup.py, then I get a different problem.

sisl-install: using internal shipped revisions
running install
running bdist_egg
running egg_info
running build_src
build_src
building extension "sisl.io.siesta._siesta" sources
f2py options: []
  adding 'build\src.win-amd64-3.7\build\src.win-amd64-3.7\sisl\io\siesta\fortranobject.c' to sources.
  adding 'build\src.win-amd64-3.7\build\src.win-amd64-3.7\sisl\io\siesta' to include_dirs.
  adding 'build\src.win-amd64-3.7\sisl\io\siesta\_siesta-f2pywrappers2.f90' to sources.
building extension "sisl._indices" sources
building extension "sisl._math_small" sources
building extension "sisl.physics._bloch" sources
building extension "sisl.physics._matrix_ddk" sources
building extension "sisl.physics._matrix_dk" sources
building extension "sisl.physics._matrix_k" sources
building extension "sisl.physics._matrix_phase3" sources
building extension "sisl.physics._matrix_phase3_nc" sources
building extension "sisl.physics._matrix_phase3_so" sources
building extension "sisl.physics._matrix_phase_nc_diag" sources
building extension "sisl.physics._matrix_phase_nc" sources
building extension "sisl.physics._matrix_phase" sources
building extension "sisl.physics._matrix_phase_so" sources
building extension "sisl.physics._phase" sources
building extension "sisl._sparse" sources
building extension "sisl._supercell" sources
build_src: building npy-pkg config files
writing sisl.egg-info\PKG-INFO
writing dependency_links to sisl.egg-info\dependency_links.txt
writing entry points to sisl.egg-info\entry_points.txt
writing requirements to sisl.egg-info\requires.txt
writing top-level names to sisl.egg-info\top_level.txt
reading manifest file 'sisl.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no directories found matching 'doc'
warning: no files found matching '*.md' under directory 'toolbox'
warning: no previously-included files found matching 'DEPLOY.md'
warning: no previously-included files matching '*.html' found anywhere in distribution
warning: no previously-included files matching '*.sh' found anywhere in distribution
warning: no previously-included files matching '.git*' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
no previously-included directories found matching 'benchmarks'
no previously-included directories found matching 'deprecated'
no previously-included directories found matching 'developments'
no previously-included directories found matching 'ci'
no previously-included directories found matching 'files'
no previously-included directories found matching '__pycache__'
writing manifest file 'sisl.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
copying sisl\info.py -> build\lib.win-amd64-3.7\sisl
running build_ext
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
customize MSVCCompiler using EnsureBuildExt
get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelVisualFCompiler
Could not locate executable ifort
Could not locate executable ifl
customize AbsoftFCompiler
Could not locate executable f90
customize CompaqVisualFCompiler
Found executable C:\Program Files\Git\usr\bin\DF.exe
Could not locate executable C:\Program
customize IntelItaniumVisualFCompiler
Could not locate executable efl
customize Gnu95FCompiler
Found executable C:\msys64\mingw64\bin\gfortran.exe
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gfortran.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.1.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++ --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --disable-plugin --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev3, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.1.0 (Rev3, Built by MSYS2 project) 
customize Gnu95FCompiler
gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gfortran.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/lto-wrapper.exe
gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.1.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++ --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --disable-plugin --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev3, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.1.0 (Rev3, Built by MSYS2 project) 
gfortran.exe: error: unrecognized command-line option '-mno-cygwin'; did you mean '-mno-clwb'?
customize Gnu95FCompiler using EnsureBuildExt
building 'sisl.io.siesta._siesta' extension
compiling C sources
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/

[zerothi]

Hmm...

Your first post was because the HDF5 library + headers were not available.

Your second post is because the gfortran compilation fails with some weird looking flags. I don't know how they entered, i.e. I can't find them in numpy which is used for the extensions.

I don't have a windows box to test this with. So I can't be of much help. I guess the fortran problem could be googleable?

[kovalp]

Dear Nick,

Thank you for your comments.

After installing pip install h5py and the microsoft build tools in the following configuration

the installation with pip install sisl finished reporting success.

However, I still could not import sisl module

koval@DESKTOP-B5CLTB9 MINGW64 ~
$ python -c "import sisl; print(sisl.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\__init__.py", line 128, in <module>
    from . import io
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\io\__init__.py", line 284, in <module>
    from .tbtrans import *
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\io\tbtrans\__init__.py", line 71, in <module>
    from .delta import *
  File "C:\Users\koval\AppData\Local\Programs\Python\Python37\lib\site-packages\sisl\io\tbtrans\delta.py", line 16, in <module>
    from ..siesta._siesta import siesta_sc_off
ImportError: DLL load failed: The specified module could not be found.
DLL load failed: The specified module could not be found.

Any hint is appreciated. I suspect the failing import is a Fortran extension (..siesta._siesta), isn't?

PS. I am using a free unregistered windows in the virtualbox to do these trials.

PPS. The Fortran compiler has been provided by MSYS2 tools. The C compiler has been provided by Visual Studio. It would be nice to convince pip install sisl to use C compiler from MSYS2.

[zerothi]

Hmmm, could you try:

pip install -vvv --global-option "build --compiler=<compiler> --fcompiler=<compiler>" sisl

where <compiler> is one of msvc9, msvc, _msvc, cygwinc.
The -vvv should give additional details on the compilation process. However, it is quite weird it successfully installs but fails to install the fortran-sources.

[zerothi]

Oh, --fcompiler=gnu should also be tried!

[kovalp]

Thank you for your comments. I had tried some of these options. I guess a minimal sisl with just one small subroutine using a fortran function would help in debugging. Whenever I use --global-option, pip recompiles cython and numpy. This takes a load of time and produces the non-working installation. I used -v. It uncovers a lot indeed, but I am lacking a better control on the compilers and on what has to be compiled. I had more control when using python setup.py install, but still no luck (because my msys2 gfortran accepts posix slashes in paths, not windows ones I think).

[zerothi]

Yeah, pip is not always easy to work with.

To better control the environment with pip you could do something like this (this command will require you have all dependencies pre-installed and reachable in PYTHONPATH):

pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=cygwinc --global-option --fcompiler=gnu95 sisl

What does this give of output?

[kovalp]

$ pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=cygwinc --global-option --fcompiler=gnu95 sisl
Defaulting to user installation because normal site-packages is not writeable
Ignoring indexes: https://pypi.org/simple
Created temporary directory: C:\Users\kovalp\AppData\Local\Temp\pip-ephem-wheel-cache-t0xsaw0a
Created temporary directory: C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Initialized build tracking at C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Created build tracker: C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Entered build tracker: C:\Users\kovalp\AppData\Local\Temp\pip-req-tracker-d_svvy0z
Created temporary directory: C:\Users\kovalp\AppData\Local\Temp\pip-install-3q197tre
0 location(s) to search for versions of sisl:
Given no hashes to check 0 links for project 'sisl': discarding no candidates
C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\commands\install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
ERROR: Could not find a version that satisfies the requirement sisl (from versions: none)
ERROR: No matching distribution found for sisl
Exception information:
Traceback (most recent call last):
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\cli\base_command.py", line 188, in _main
    status = self.run(options, args)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\cli\req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\commands\install.py", line 333, in run
    reqs, check_supported_wheels=not options.target_dir
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 313, in _get_abstract_dist_for
    self._populate_link(req)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\resolution\legacy\resolver.py", line 279, in _populate_link
    req.link = self.finder.find_requirement(req, upgrade)
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\pip\_internal\index\package_finder.py", line 930, in find_requirement
    req)
pip._internal.exceptions.DistributionNotFound: No matching distribution found for sisl
Removed build tracker: 'C:\\Users\\kovalp\\AppData\\Local\\Temp\\pip-req-tracker-d_svvy0z'

[zerothi]

Ah, yes, download the sisl.tar.gz, extract, cd, and do pip .... . (i.e. replace sisl with .)

[kovalp]

pip.out.txt

I know how to solve that. I will solve. However, the problem is that it starts to ask about visual studio C/C++.

cygwinc I don't have.

[zerothi]

Hmm, I don't know what this means, but this seems like a general case.. I didn't get if you know what to do, or if you still need help ;)

It seems this is the correct direction?

[kovalp]

This particular trouble I can solve (by installing Visual C). I will see what comes later. I still use your experience on this.

[kovalp]

Below is the output. The installation is successful. pip.out.txt

However, I still experience the same trouble when importing sisl

kovalp@DESKTOP-AT1KGE5 MINGW64 ~/Downloads/sisl-master/sisl-master
$ pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=gcc --global-option --fcompiler=gnu95 . &>pip.out

kovalp@DESKTOP-AT1KGE5 MINGW64 ~/Downloads/sisl-master/sisl-master
$ cd

kovalp@DESKTOP-AT1KGE5 MINGW64 ~
$ python -c "import sisl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\__init__.py", line 128, in <module>
    from . import io
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\io\__init__.py", line 284, in <module>
    from .tbtrans import *
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\io\tbtrans\__init__.py", line 71, in <module>
    from .delta import *
  File "C:\Users\kovalp\AppData\Roaming\Python\Python37\site-packages\sisl\io\tbtrans\delta.py", line 16, in <module>
    from ..siesta._siesta import siesta_sc_off
ImportError: DLL load failed: No se puede encontrar el m�dulo especificado.
DLL load failed: No se puede encontrar el m�dulo especificado.

[zerothi]

Hmm, I must admit I have no clue... It seems that the fortran code compiled does not work correctly.

It is also unclear to me what the importance of the compilers are. You seem to be using msys64 for fortran sources, but MSVC for c-code, so basically you are compiling with one compiler and linking with another. This could be the reason.

Could you try with

$ pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=mingw32 --global-option --fcompiler=gnu95 . &>pip.out

Or try something that changes the C-compiler. Play around with --compiler options I think could be a test, check in the output how it does the final linking. In your output, this line is the key:

  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /nodefaultlib:libucrt.lib ucrt.lib /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\msys64\mingw64\lib\gcc\x86_64-w64-mingw32\9.1.0 /LIBPATH:c:\program files\python37\libs /LIBPATH:c:\program files\python37\PCbuild\amd64 /LIBPATH:c:\program files\python37\libs /LIBPATH:c:\program files\python37\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64 /EXPORT:PyInit__siesta build\temp.win-amd64-3.7\Release\build\src.win-amd64-3.7\sisl\io\siesta\_siestamodule.obj build\temp.win-amd64-3.7\Release\build\src.win-amd64-3.7\build\src.win-amd64-3.7\sisl\io\siesta\fortranobject.obj build\temp.win-amd64-3.7\Release\libio_m.DL5GA7BPLQZXMPFPWEFJ2SIMXZB6UBNR.gfortran-win_amd64.lib /OUT:build\lib.win-amd64-3.7\sisl\io\siesta\_siesta.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\build\src.win-amd64-3.7\sisl\io\siesta\_siesta.cp37-win_amd64.lib

I.e. the module _siesta.cp...

[kovalp]

Thank you for your comments. The problem is that it is difficult to control the C compiler and there is no Fortran compiler provided by the visual studio.

[zerothi]

Hmm. the flag --global-option --compiler=<> controls the C-compiler.

[kovalp]

I already specifying --compiler=gcc, but the pip machinery still uses visual studio's C.

[zerothi]

I said you should try mingw32? Could you just try out the different options...

 f2py -c --help-compiler                                                  
List of available compilers:
  --compiler=bcpp      Borland C++ Compiler
  --compiler=cygwin    Cygwin port of GNU C Compiler for Win32
  --compiler=intel     Intel C Compiler for 32-bit applications
  --compiler=intele    Intel C Itanium Compiler for Itanium-based applications
  --compiler=intelem   Intel C Compiler for 64-bit applications
  --compiler=intelemw  Intel C Compiler for 64-bit applications on Windows
  --compiler=intelw    Intel C Compiler for 32-bit applications on Windows
  --compiler=mingw32   Mingw32 port of GNU C Compiler for Win32
  --compiler=msvc      Microsoft Visual C++
  --compiler=pathcc    PathScale Compiler for SiCortex-based applications
  --compiler=unix      standard UNIX-style compiler
Removing build directory /tmp/tmpslbczsoh

Nevermind that mingw32 refers to 32-bit, it is handled correctly internally (at least it should).

[kovalp]

I investigated that already. The option --compiler has no effect in my case.

[kovalp]

What would be a minimal example to reproduce this trouble? Isn't it using any Fortran source code from Python?

[zerothi]

hmmm. ok... Then I am clueless. Other people more experienced with the windows + fortran sources could be more helpful.

Yeah, I guess a minimal example of a simple fortran source, compiled in the way I add the sources would be usable as a test case.

I.e. something that uses from numpy.distutils.core import Extension as the backend for the compilation.

[kovalp]

Could you kindly provide such a "Hello world" example?

[zerothi]

I think the problem comes from the full setup.py and package process.

You could try with a simple code:

hello_world.f90

subroutine hello_world()
print *,"tshetsh"
end

And a setup.py looking something like:

#!/usr/bin/env python3

import sys
import subprocess
import multiprocessing
import os
import os.path as osp
import argparse

# pkg_resources are part of setuptools
import pkg_resources

# We should *always* import setuptools prior to Cython/distutils
import setuptools

min_version ={
    "numpy": "1.13",
}

# Although setuptools is not shipped with the standard library, I think
# this is ok since it should get installed pretty easily.
from setuptools import Command, Extension
from setuptools import find_packages

# Patch to allow fortran sources in setup
# build_ext requires numpy setup
# Also for extending build schemes we require build_ext from numpy.distutils
from distutils.command.sdist import sdist
from numpy.distutils.command.build_ext import build_ext as numpy_build_ext
from numpy.distutils.core import Extension as FortranExtension
from numpy.distutils.core import setup

# Custom command classes
cmdclass = {}

suffix = ".c"

# Retrieve the compiler information
from numpy.distutils.system_info import get_info
# use flags defined in numpy
all_info = get_info('ALL')

cmdclass["sdist"] = sdist

extensions = []
# Specific Fortran extensions
ext_fortran = {
    "package1._sources": {
        "sources": [f"package1/_src/{f}" for f in
                    ("hello_world.f90")
        ],
    },
}

for name, data in ext_fortran.items():
    ext = FortranExtension(name,
        sources=data.get("sources"),
        depends=data.get("depends", []),
        include_dirs=data.get("include", None),
        define_macros=macros + data.get("macros", []),
    )

    extensions.append(ext)

# Override build_ext command (typically called by setuptools)
cmdclass["build_ext"] = numpy_build_ext

# The install_requires should also be the
# requirements for the actual running of sisl
setuptools_kwargs = {
    "install_requires": [
        "setuptools",
        "numpy >= " + min_version["numpy"],
    ],
    "setup_requires": [
        "numpy >= " + min_version["numpy"],
    ],
}

metadata = dict(
    name="package1",
    maintainer="foo",
    description="bar",
    long_description="snthaoe"
    long_description_content_type="text/markdown",
    packages=
    # We need to add sisl.* since that recursively adds modules
    find_packages(include=["package1", "package1.*"])
    ext_modules=extensions,
    cmdclass=cmdclass,
    **setuptools_kwargs
)

if __name__ == "__main__":

    metadata["version"] = "1.0"

    # Freeze to support parallel compilation when using spawn instead of fork
    multiprocessing.freeze_support()
    setup(**metadata)

Then create the small package, (here just named package1) and put in some boiler plate code.

[kovalp]

Thank you very much!

[zerothi]

I hope this guides you in the right direction.

Let me advice you to do this on Unix first! ;)

[kovalp]

Right. It would be nice to have a demonstrably working example.

[kovalp]

I created this

https://github.com/kovalp/sisl_approach_f2py

It does not compile.

pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=unix --global-option --fcompiler=gnu95 .

pip install -vvv --no-cache-dir --no-deps --no-index --no-build-isolation --compile --global-option build --global-option --compiler=unix --global-option --fcompiler=gnu95 .
/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/commands/install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Non-user install because user site-packages disabled
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-8s0klx5z
Created temporary directory: /tmp/pip-req-tracker-3bv0x5nf
Initialized build tracking at /tmp/pip-req-tracker-3bv0x5nf
Created build tracker: /tmp/pip-req-tracker-3bv0x5nf
Entered build tracker: /tmp/pip-req-tracker-3bv0x5nf
Created temporary directory: /tmp/pip-install-tasnr2ha
Processing /home/kovalp/programs/sisl_approach_f2py
  Created temporary directory: /tmp/pip-req-build-htz15rv7
  Added file:///home/kovalp/programs/sisl_approach_f2py to build tracker '/tmp/pip-req-tracker-3bv0x5nf'
    Running setup.py (path:/tmp/pip-req-build-htz15rv7/setup.py) egg_info for package from file:///home/kovalp/programs/sisl_approach_f2py
    Created temporary directory: /tmp/pip-pip-egg-info-gc1_rain
    Running command python setup.py egg_info
    running egg_info
    running build_src
    build_src
    building extension "package1._sources" sources
    build_src: building npy-pkg config files
    creating /tmp/pip-pip-egg-info-gc1_rain/package1.egg-info
    writing /tmp/pip-pip-egg-info-gc1_rain/package1.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-gc1_rain/package1.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-gc1_rain/package1.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-gc1_rain/package1.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-gc1_rain/package1.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-gc1_rain/package1.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-gc1_rain/package1.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-htz15rv7 has version 1.0, which satisfies requirement package1==1.0 from file:///home/kovalp/programs/sisl_approach_f2py
  Removed package1==1.0 from file:///home/kovalp/programs/sisl_approach_f2py from build tracker '/tmp/pip-req-tracker-3bv0x5nf'
Skipping wheel build for package1, due to binaries being disabled for it.
Installing collected packages: package1
  Attempting uninstall: package1
    Found existing installation: package1 1.0
    Not uninstalling package1 at /home/kovalp/programs/sisl_approach_f2py, outside environment /home/kovalp/envs/py37-pyside2
    Can't uninstall 'package1'. No files were found to uninstall.
  Created temporary directory: /tmp/pip-record-rk7j__sm
    Running command /home/kovalp/envs/py37-pyside2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-htz15rv7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-htz15rv7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build --compiler=unix --fcompiler=gnu95 install --record /tmp/pip-record-rk7j__sm/install-record.txt --single-version-externally-managed --compile --install-headers /home/kovalp/envs/py37-pyside2/include/site/python3.7/package1
    running build
    running config_cc
    unifing config_cc, config, build_clib, build_ext, build commands --compiler options
    running config_fc
    unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
    running build_src
    build_src
    building extension "package1._sources" sources
    build_src: building npy-pkg config files
    running build_ext
    customize UnixCCompiler
    customize UnixCCompiler using build_ext
    building 'package1._sources' extension
    compiling C sources
    C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

    error: unknown file type '' (from 'package1/_src/h')
    Running setup.py install for package1 ... error
  ERROR: Can't roll back package1; was not uninstalled
ERROR: Command errored out with exit status 1: /home/kovalp/envs/py37-pyside2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-htz15rv7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-htz15rv7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build --compiler=unix --fcompiler=gnu95 install --record /tmp/pip-record-rk7j__sm/install-record.txt --single-version-externally-managed --compile --install-headers /home/kovalp/envs/py37-pyside2/include/site/python3.7/package1 Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 841, in install
    req_description=str(self.req),
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/operations/install/legacy.py", line 86, in install
    raise LegacyInstallFailure
pip._internal.operations.install.legacy.LegacyInstallFailure

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 407, in run
    use_user_site=options.use_user_site,
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
    **kwargs
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 845, in install
    six.reraise(*exc.parent)
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_vendor/six.py", line 703, in reraise
    raise value
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/operations/install/legacy.py", line 76, in install
    cwd=unpacked_source_directory,
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 274, in runner
    spinner=spinner,
  File "/home/kovalp/envs/py37-pyside2/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 241, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/kovalp/envs/py37-pyside2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-htz15rv7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-htz15rv7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build --compiler=unix --fcompiler=gnu95 install --record /tmp/pip-record-rk7j__sm/install-record.txt --single-version-externally-managed --compile --install-headers /home/kovalp/envs/py37-pyside2/include/site/python3.7/package1 Check the logs for full command output.
Removed build tracker: '/tmp/pip-req-tracker-3bv0x5nf'
(py37-pyside2) kovalp:sisl_approach_f2py$ 

This hello world example assumes a bit more knowledge than I have.

[kovalp]

A temporary solution would be add a [build] section to setup.cfg as following

koval@DESKTOP-B5CLTB9 MINGW64 ~/programs/sisl (master)
$ head setup.cfg
[build]
compiler=mingw32
fcompiler=gfortran

in this case, doing python setup.py build in the source directory, followed by python setup.py install resutls in a valid sisl installation.

$ python -c "import sisl; print(sisl)"
<module 'sisl' from 'C:\\Users\\koval\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\sisl-0.10.0-py3.7-win-amd64.egg\\sisl\\__init__.py'>

The Visual Studio is not necessary in a clean install I bet.

[kovalp]

Shouldn't we produce a binary version of SISL in this way?

[kovalp]

The instructions to get the sisl with msys2 compiler could be following

Install MSYS2, install gcc/gfortran in MSYS2. Add MSYS2 bin directory to PATH. Install a common Python (not Anaconda, not Python in MSYS2).

Using pip Perhaps pip install --upgrade pip then pip install --upgrade cython h5py netCDF4 cftime tqdm numpy scipy matplotlib

git clone https://github.com/zerothi/sisl.git
cd sisl

Edit setup.cfg adding a section [build]

[build]
compiler=mingw32
fcompiler=gfortran

python setup.py build
python setup.py install

setup.build.out.txt setup.install.out.txt

Testing is not very successful yet

Step outside the source directory

cd
pytest --pyargs -x sisl &>pytest.out.txt

pytest.out.txt

[zerothi]

A temporary solution would be add a [build] section to setup.cfg as following

koval@DESKTOP-B5CLTB9 MINGW64 ~/programs/sisl (master)
$ head setup.cfg
[build]
compiler=mingw32
fcompiler=gfortran

in this case, doing python setup.py build in the source directory, followed by python setup.py install resutls in a valid sisl installation.

$ python -c "import sisl; print(sisl)"
<module 'sisl' from 'C:\\Users\\koval\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\sisl-0.10.0-py3.7-win-amd64.egg\\sisl\\__init__.py'>

The Visual Studio is not necessary in a clean install I bet.

Great thanks for the report!

[zerothi]

Shouldn't we produce a binary version of SISL in this way?

Any merge request is most welcome!!! :+1: I don't know where it should be stored etc. you could play with conda enviroment, given it now works. But I don't know how.

[zerothi]

Testing is not very successful yet

Step outside the source directory

cd
pytest --pyargs -x sisl &>pytest.out.txt

pytest.out.txt

appdata\local\programs\python\python37\lib\importlib\_bootstrap.py:219
appdata\local\programs\python\python37\lib\importlib\_bootstrap.py:219
  c:\users\koval\appdata\local\programs\python\python37\lib\importlib\_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
    return f(*args, **kwds)

I believe this is the culprit. This suggests to me that sisl was built against a different numpy vs the one you are using; or at least something is not consistent with your setup.
I would try with a clean environment so that you are sure you are not mixing numpy versions etc.

Also, could you download the latest sisl commit, there are some changes in the test-handling to ensure pathlib.Path usage, perhaps this is one cause?

[kovalp]

Shouldn't we produce a binary version of SISL in this way?

Any merge request is most welcome!!! 👍 I don't know where it should be stored etc. you could play with conda enviroment, given it now works. But I don't know how.

I think Anaconda package could be useful indeed. However, this is not what we regularly use. So, I can't say "when".

[kovalp]

Testing is not very successful yet Step outside the source directory

cd
pytest --pyargs -x sisl &>pytest.out.txt

pytest.out.txt

appdata\local\programs\python\python37\lib\importlib\_bootstrap.py:219
appdata\local\programs\python\python37\lib\importlib\_bootstrap.py:219
  c:\users\koval\appdata\local\programs\python\python37\lib\importlib\_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
    return f(*args, **kwds)

I believe this is the culprit. This suggests to me that sisl was built against a different numpy vs the one you are using; or at least something is not consistent with your setup. I would try with a clean environment so that you are sure you are not mixing numpy versions etc.

Also, could you download the latest sisl commit, there are some changes in the test-handling to ensure pathlib.Path usage, perhaps this is one cause?

I saw these binary incompatibility warnings, but they seem to be harmless.

I redid the testing with new version of sisl with similar result pytest.out.txt

[kovalp]

This wheel binary distribution is created for latest sisl and Python 3.7

https://www.dropbox.com/s/8r23d8k9yqnberk/sisl-0.10.0-cp37-cp37m-win_amd64.whl?dl=1

This should be useful for installation without compilation

pip install sisl-0.10.0-cp37-cp37m-win_amd64.whl

I used this blog to produce the wheel file. https://dzone.com/articles/executable-package-pip-install

[zerothi]

Could you instead run

pytest --pyargs -rX sisl

You force it to stop on the first error...

Also, could you try the latest commit, I hope this one fixes it ;)

[kovalp]

What does the option -r mean? Using the options -rX would not stop after first error, so I used

pytest --pyargs sisl -x &>pytest.out.txt

It got farther (((b)))

pytest.out.txt

Another question is bothering me. Why would some test skip?

appdata\local\programs\python\python37\lib\site-packages\sisl-0.10.0-py3.7-win-amd64.egg\sisl\io\siesta\tests\test_bands.py s [  0%]
ss                                                                       [  1%]

I looked at them, but did not find the reason for skipping.

[zerothi]

What does the option -r mean? Using the options -rX would not stop after first error, so I used

See pytest --help:

  -r chars              show extra test summary info as specified by chars:
                        (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed,
                        (p)assed, (P)assed with output, (a)ll except passed
                        (p/P), or (A)ll. (w)arnings are enabled by default (see
                        --disable-warnings), 'N' can be used to reset the list.
                        (default: 'fE').

pytest --pyargs sisl -x &>pytest.out.txt

It got farther (((b)))

pytest.out.txt

Great, it seems to me that there is still some binary mismatch which causes it to fail... :+1:

Another question is bothering me. Why would some test skip?

appdata\local\programs\python\python37\lib\site-packages\sisl-0.10.0-py3.7-win-amd64.egg\sisl\io\siesta\tests\test_bands.py s [  0%]
ss                                                                       [  1%]

I looked at them, but did not find the reason for skipping.

Some tests skip because of missing (optional) modules (matplotlib, xarray), others because a 2nd set of files are needed for extensive tests (the files). These files are not in the repository since it would bloat the repo with huge files.

[zerothi]

Did you try the latest commit? Also, yet another commit has just been made to ensure paths in setup.py are windows friendly, however, this shouldn't interfere as your code compiled fine... Hmm... The machinery for windows is really weird, borderline unix... But not quite.

[kovalp]

Ok, I understand that you wanted to try the whole test suite

Here you go pytest.out.txt

[zerothi]

Thanks.

I think this looks ok. Except for the fact that all netcdf tests where it is writing, fails...

• test_get_dtype1 is reasonable since windows and unix have different default int values
• test_ordered2 is somewhat arbitrary and probably the sleep command on windows does not work (not important test)
• test_create_1d_bcasting_data_2d is a bit worrying!

[kovalp]

To netcdf, I think the temporary directory is not created at the momemt of file creation, therefore the trouble. I produced a script out of the first failing test and it works fine, producing a .nc file as expected.

import numpy as np
from sisl import Atom, Geometry, DensityMatrix, SuperCell, Hamiltonian
# from sisl import DynamicalMatrix
from sisl.io.siesta import ncSileSiesta

def sisl_system_funct():
    """ A preset list of geometries/Hamiltonians. """
    class System:
        pass

    d = System()

    alat = 1.42
    sq3h = 3.**.5 * 0.5
    C = Atom(Z=6, R=1.42)
    sc = SuperCell(np.array([[1.5, sq3h, 0.],
                             [1.5, -sq3h, 0.],
                             [0., 0., 10.]], np.float64) * alat,
                   nsc=[3, 3, 1])
    d.g = Geometry(np.array([[0., 0., 0.],
                             [1., 0., 0.]], np.float64) * alat,
                   atoms=C, sc=sc)

    d.R = np.array([0.1, 1.5])
    d.t = np.array([0., 2.7])
    d.tS = np.array([(0., 1.0),
                     (2.7, 0.)])
    d.C = Atom(Z=6, R=max(d.R))
    d.sc = SuperCell(np.array([[1.5, sq3h, 0.],
                               [1.5, -sq3h, 0.],
                               [0., 0., 10.]], np.float64) * alat,
                     nsc=[3, 3, 1])
    d.gtb = Geometry(np.array([[0., 0., 0.],
                               [1., 0., 0.]], np.float64) * alat,
                     atoms=C, sc=sc)

    d.ham = Hamiltonian(d.gtb)
    d.ham.construct([(0.1, 1.5), (0.1, 2.7)])
    return d

def test_nc_multiple_fail():
    # writing two different sparse matrices to the same
    # file will fail
    sisl_system = sisl_system_funct()
    f = "filename.nc"
    H = Hamiltonian(sisl_system.gtb)
    DM = DensityMatrix(sisl_system.gtb)

    sile = ncSileSiesta(f, 'w')
    H.construct([sisl_system.R, sisl_system.t])
    H.write(sile)

if __name__ == "__main__":
    test_nc_multiple_fail()

[kovalp]

How to run sisl from within the source? Uninstall the pip package and add the source directory to the pythonpath?

[zerothi]

Hmm, that is weird.

The directory-creation method is used in nearly all tests that deal with io, so it is very weird. It seems that sisl still works properly.

I guess you can't run the sisl source from the source directory since it requires C and fortran sources build to a library. If you want to play with changing stuff without re-installing, then just change files in the installed directory, and delete the corresponding file in the same __pycache__ directory. That should work.

[zerothi]

If you still want to run in-source you could do:

pip3 install -e .

or something similar.

[kovalp]

Thank you!

In order to fix the issue, I would need to find a way to install without changing setup.cfg, I guess. However, I still didn't find the command for doing it under windows. The change in setup.cfg seems to affect the installation differently than the command-line options.

[zerothi]

In order to fix the issue, I would need to find a way to install without changing setup.cfg, I guess. However, I still didn't find the command for doing it under windows. The change in setup.cfg seems to affect the installation differently than the command-line options.

Agreed, this seems like a shortcoming on windows. Could you try and put the build section in this file %HOME%\pydistutils.cfg, see here.

[zhyphy]

Dear All， I just successfully installed SISL on the win10 with anaconda3 / vs2015 / intel2017. Use “python setup.py install” instead fo "pip install sisl"

[kovalp]

Good to know. Thank you very much!