search

conda-forge / autotools_clang_conda-feedstock

A conda-smithy repository for autotools_clang_conda.
BSD 3-Clause "New" or "Revised" License
5 stars 6 forks source link

autotools_clang_conda + openmp + numpy + Windows = dependency hell #11

Open tschoonj opened 4 years ago

tschoonj commented 4 years ago

Hi @isuruf

I am currently in the process of developing a Windows build for polycap's conda-forge recipe. @PieterTack and myself are the developers of polycap.

This software package uses GNU autotools as build system, and has xraylib (one of my personal projects) , hdf5 and gsl as dependencies, and needs an OpenMP enabled C compiler. It also comes with Python bindings that have been written using Cython, and uses the Numpy C API.

During my initial attempts at getting the polycap Windows build up and running, I tried to emulate what I do in the xraylib feedstock: use the MinGW-W64 compilers to do the heavy lifting, combined with gendef and dlltool to generate the GNU ld compatible import libraries.

This got me quite far, but eventually I ran into trouble with GSL and hdf5, which are both using global variables that kept producing linking errors, no matter what I tried to do...

Looking at the gsl feedstock I noticed your use of autotools_clang_conda, which I gave a try. Initially I did so with xraylib, to get the proper xrl.lib import library, but requiring a lot of changes in its codebase as I depended on POSIX extensions that clang on Windows does not support as they are not present in msvcrt.

With this working, I tried the same approach for polycap, and got rid of all the extensions that would have torpedoed the build with clang.

However, when trying to build this on my Windows 10 VM, I quickly descend into dependency hell:

``` $ conda build -c conda-forge -c defaults . No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11 WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11 Adding in variants from internal_defaults INFO:conda_build.variants:Adding in variants from internal_defaults Attempting to finalize metadata for polycap INFO:conda_build.metadata:Attempting to finalize metadata for polycap Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done BUILD START: ['polycap-1.0-py37h8c2d366_1.tar.bz2'] Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done ## Package Plan ## environment location: C:\Users\TomSc\miniconda3\conda-bld\polycap_1592216523502\_h_env The following NEW packages will be INSTALLED: ca-certificates: 2020.4.5.2-hecda079_0 conda-forge certifi: 2020.4.5.2-py37hc8dfbb8_0 conda-forge cython: 0.29.20-py37h1834ac0_0 conda-forge gsl: 2.6-h86b8168_0 conda-forge hdf5: 1.10.6-nompi_ha405e13_100 conda-forge intel-openmp: 2020.0-166 libblas: 3.8.0-15_mkl conda-forge libcblas: 3.8.0-15_mkl conda-forge liblapack: 3.8.0-15_mkl conda-forge llvm-openmp: 10.0.0-0 conda-forge mkl: 2020.0-166 numpy: 1.11.3-py37h873a0b8_1207 conda-forge openssl: 1.1.1g-he774522_0 conda-forge pip: 20.1.1-py_1 conda-forge python: 3.7.6-cpython_h60c2a47_6 conda-forge python_abi: 3.7-1_cp37m conda-forge setuptools: 47.1.1-py37hc8dfbb8_0 conda-forge sqlite: 3.30.1-hfa6e2cd_0 conda-forge vc: 14.1-h869be7e_1 conda-forge vs2015_runtime: 14.16.27012-h30e32a0_2 conda-forge wheel: 0.34.2-py_1 conda-forge wincertstore: 0.2-py37_1003 conda-forge xraylib: 4.0.0-py37hf9181ef_2 local zlib: 1.2.11-h2fa13f4_1006 conda-forge Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... WARNING:conda.models.version:Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.*, but conda is ignoring the .* and treating it as 1 WARNING conda.models.version:get_matcher(531): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.*, but conda is ignoring the .* and treating it as 1 WARNING:conda.models.version:Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 2.*, but conda is ignoring the .* and treating it as 2 WARNING conda.models.version:get_matcher(531): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 2.*, but conda is ignoring the .* and treating it as 2 WARNING:conda.models.version:Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 3.*, but conda is ignoring the .* and treating it as 3 WARNING conda.models.version:get_matcher(531): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 3.*, but conda is ignoring the .* and treating it as 3 WARNING:conda.models.version:Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 4.*, but conda is ignoring the .* and treating it as 4 WARNING conda.models.version:get_matcher(531): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 4.*, but conda is ignoring the .* and treating it as 4 WARNING:conda.models.version:Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 2.*, but conda is ignoring the .* and treating it as 2 WARNING conda.models.version:get_matcher(531): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 2.*, but conda is ignoring the .* and treating it as 2 WARNING:conda.models.version:Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 0.23.*, but conda is ignoring the .* and treating it as 0.23 WARNING conda.models.version:get_matcher(531): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 0.23.*, but conda is ignoring the .* and treating it as 0.23 WARNING:conda.models.version:Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 3.6.*, but conda is ignoring the .* and treating it as 3.6 WARNING conda.models.version:get_matcher(531): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 3.6.*, but conda is ignoring the .* and treating it as 3.6 Found conflicts! Looking for incompatible packages. This can take several minutes. Press CTRL-C to abort. failed failed to parse packages from exception: The following specifications were found to be incompatible with each other: Output in format: Requested package -> Available versions Package openblas conflicts for: xraylib[version='>=4.0.0'] -> numpy[version='>=1.11.3,<2.0a0'] -> openblas[version='0.2.20|0.2.20.*|>=0.2.20,<0.2.21.0a0|>=0.3.3,<0.3.4.0a0'] numpy[version='>=1.11.3,<2.0a0'] -> libblas[version='>=3.8.0,<4.0a0'] -> openblas[version='0.3.5.*|0.3.6|>=0.3.6,<0.3.7.0a0',build=h828a276_2] gsl[version='>=2.6,<2.7.0a0'] -> libblas[version='>=3.8.0,<4.0a0'] -> openblas[version='0.3.5.*|0.3.6|>=0.3.6,<0.3.7.0a0',build=h828a276_2] numpy[version='>=1.11.3,<2.0a0'] -> openblas[version='0.2.20|0.2.20.*|>=0.2.20,<0.2.21.0a0|>=0.3.3,<0.3.4.0a0'] Package libblas conflicts for: gsl[version='>=2.6,<2.7.0a0'] -> libcblas[version='>=3.8.0,<4.0a0'] -> libblas[version='3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0.*',build='4_blis|4_openblas|4_mkl|5_blis|5_mkl|7_blis|7_mkl|7_openblas|8_mkl|9_openblas|10_openblas|10_mkl|11_blis|11_mkl|12_blis|12_openblas|13_blis|14_blis|15_openblas|16_openblas|15_mkl|15_blis|14_openblas|14_mkl|13_openblas|13_mkl|12_mkl|11_openblas|10_blis|9_mkl|9_blis|8_openblas|8_blis|6_mkl|6_openblas|6_blis|5_openblas'] gsl[version='>=2.6,<2.7.0a0'] -> libblas[version='>=3.8.0,<4.0a0'] Package vs2015_runtime conflicts for: zlib -> vc[version='>=14.1,<15.0a0'] -> vs2015_runtime[version='>=14.0.25123,<15.0a0|>=14.0.25420|>=14.15.26706|>=14.16.27012'] numpy[version='>=1.11.3,<2.0a0'] -> python[version='>=3.6,<3.7.0a0'] -> vs2015_runtime[version='>=14.0.25123,<15.0a0|>=14.0.25420|>=14.15.26706'] numpy[version='>=1.11.3,<2.0a0'] -> vs2015_runtime[version='>=14.16.27012|>=14.16.27012,<15.0a0'] llvm-openmp[version='>=10.0.0'] -> vs2015_runtime[version='>=14.16.27012'] xraylib[version='>=4.0.0'] -> vs2015_runtime[version='>=14.16.27012'] xraylib[version='>=4.0.0'] -> numpy[version='>=1.11.3,<2.0a0'] -> vs2015_runtime[version='>=14.15.26706|>=14.16.27012,<15.0a0'] python_abi=3.7[build=*_cp37m] -> python=3.7 -> vs2015_runtime[version='>=14.16.27012|>=14.16.27012,<15.0a0'] vc[version='>=14.1,<15.0a0'] -> vs2015_runtime[version='>=14.15.26706|>=14.16.27012'] vs2015_runtime[version='>=14.16.27012'] zlib -> vs2015_runtime[version='>=14.16.27012,<15.0a0'] llvm-openmp[version='>=10.0.0'] -> vc[version='>=14.1,<15.0a0'] -> vs2015_runtime[version='>=14.15.26706'] python[version='>=3.7,<3.8.0a0'] -> vc[version='>=14.1,<15.0a0'] -> vs2015_runtime[version='>=14.0.25123,<15.0a0|>=14.0.25420|>=14.15.26706'] gsl[version='>=2.6,<2.7.0a0'] -> vc[version='>=14.1,<15.0a0'] -> vs2015_runtime[version='>=14.15.26706|>=14.16.27012'] python[version='>=3.7,<3.8.0a0'] -> vs2015_runtime[version='>=14.16.27012|>=14.16.27012,<15.0a0'] hdf5[version='>=1.10.6,<1.10.7.0a0'] -> vc[version='>=14.1,<15.0a0'] -> vs2015_runtime[version='>=14.15.26706|>=14.16.27012|>=14.16.27012,<15.0a0'] Package zlib conflicts for: zlib hdf5[version='>=1.10.6,<1.10.7.0a0'] -> zlib[version='>=1.2.11,<1.3.0a0'] python[version='>=3.7,<3.8.0a0'] -> sqlite[version='>=3.30.1,<4.0a0'] -> zlib[version='>=1.2.11,<1.3.0a0'] Package ca-certificates conflicts for: numpy[version='>=1.11.3,<2.0a0'] -> python[version='>=2.7,<2.8.0a0'] -> ca-certificates xraylib[version='>=4.0.0'] -> python[version='>=2.7,<2.8.0a0'] -> ca-certificates python[version='>=3.7,<3.8.0a0'] -> openssl[version='>=1.1.1g,<1.1.2a'] -> ca-certificates Package vc conflicts for: python[version='>=3.7,<3.8.0a0'] -> vc[version='14.*|>=14.1,<15.0a0|>=14,<15.0a0'] hdf5[version='>=1.10.6,<1.10.7.0a0'] -> vc[version='>=14.1,<15.0a0'] llvm-openmp[version='>=10.0.0'] -> vc[version='>=14.1,<15.0a0'] python[version='>=3.7,<3.8.0a0'] -> sqlite[version='>=3.30.1,<4.0a0'] -> vc=9 numpy[version='>=1.11.3,<2.0a0'] -> vc[version='14.*|9.*|>=14,<15.0a0|>=14.1,<15.0a0'] hdf5[version='>=1.10.6,<1.10.7.0a0'] -> zlib[version='>=1.2.11,<1.3.0a0'] -> vc[version='14.*|9.*|>=14,<15.0a0|>=9,<10.0a0'] zlib -> vc[version='10.*|14.*|9.*|>=14,<15.0a0|>=14.1,<15.0a0|>=9,<10.0a0'] gsl[version='>=2.6,<2.7.0a0'] -> vc[version='>=14.1,<15.0a0'] xraylib[version='>=4.0.0'] -> vc[version='>=14.1,<15.0a0'] vc[version='>=14.1,<15.0a0'] python_abi=3.7[build=*_cp37m] -> python=3.7 -> vc[version='14.*|>=14.1,<15.0a0|>=14,<15.0a0'] numpy[version='>=1.11.3,<2.0a0'] -> python[version='>=2.7,<2.8.0a0'] -> vc[version='14|>=9,<10.0a0'] xraylib[version='>=4.0.0'] -> numpy[version='>=1.11.3,<2.0a0'] -> vc[version='14.*|9.*|>=14,<15.0a0|>=9,<10.0a0'] Package blis conflicts for: gsl[version='>=2.6,<2.7.0a0'] -> libblas[version='>=3.8.0,<4.0a0'] -> blis[version='0.5.1.*|0.5.2|0.6.0|0.6.1|>=0.5.2,<0.5.3.0a0',build=h2fa13f4_0] numpy[version='>=1.11.3,<2.0a0'] -> libblas[version='>=3.8.0,<4.0a0'] -> blis[version='0.5.1.*|0.5.2|0.6.0|0.6.1|>=0.5.2,<0.5.3.0a0',build=h2fa13f4_0] Package numpy conflicts for: xraylib[version='>=4.0.0'] -> numpy[version='>=1.11.3,<2.0a0|>=1.14.6,<2.0a0'] numpy[version='>=1.11.3,<2.0a0'] Package python conflicts for: xraylib[version='>=4.0.0'] -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0'] numpy[version='>=1.11.3,<2.0a0'] -> python_abi=3.8[build=*_cp38] -> python[version='2.7.*|3.7.*|3.8.*'] xraylib[version='>=4.0.0'] -> numpy[version='>=1.11.3,<2.0a0'] -> python[version='3.5.*|3.6.*|>=3.5,<3.6.0a0|3.7.*|3.8.*'] numpy[version='>=1.11.3,<2.0a0'] -> python[version='3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0'] python_abi=3.7[build=*_cp37m] -> python=3.7 python[version='>=3.7,<3.8.0a0'] Package libopenblas conflicts for: libopenblas gsl[version='>=2.6,<2.7.0a0'] -> libblas[version='>=3.8.0,<4.0a0'] -> libopenblas[version='0.3.6|0.3.7|0.3.7|0.3.8|0.3.9',build='h29e5d5d_0|hcfe7411_0|h29e5d5d_1|h29e5d5d_6'] numpy[version='>=1.11.3,<2.0a0'] -> libblas[version='>=3.8.0,<4.0a0'] -> libopenblas[version='0.3.6|0.3.7|0.3.7|0.3.8|0.3.9',build='h29e5d5d_0|hcfe7411_0|h29e5d5d_1|h29e5d5d_6'] Package python_abi conflicts for: python_abi=3.7[build=*_cp37m] numpy[version='>=1.11.3,<2.0a0'] -> python_abi[version='3.6.*|3.8.*|3.7.*',build='*_cp37m|*_cp38|*_cp36m'] xraylib[version='>=4.0.0'] -> python_abi[version='3.6.*|3.7.*|3.8.*',build='*_cp37m|*_cp36m|*_cp38'] Package vs2008_runtime conflicts for: xraylib[version='>=4.0.0'] -> python[version='>=2.7,<2.8.0a0'] -> vs2008_runtime zlib -> vc[version='>=9,<10.0a0'] -> vs2008_runtime[version='>=9.0.30729.1,<10.0a0'] numpy[version='>=1.11.3,<2.0a0'] -> python[version='>=2.7,<2.8.0a0'] -> vs2008_runtime[version='>=9.0.30729.1,<10.0a0'] Package openmp conflicts for: numpy[version='>=1.11.3,<2.0a0'] -> libflang[version='>=5.0.0'] -> openmp==5.0.0 libopenblas -> libflang[version='>=5.0.0'] -> openmp==5.0.0 Package libflang conflicts for: numpy[version='>=1.11.3,<2.0a0'] -> libflang[version='>=5.0.0'] numpy[version='>=1.11.3,<2.0a0'] -> openblas[version='>=0.2.20,<0.2.21.0a0'] -> libflang libopenblas -> libflang[version='>=5.0.0'] xraylib[version='>=4.0.0'] -> numpy[version='>=1.11.3,<2.0a0'] -> libflang[version='>=5.0.0'] ```

(Apologies for the slightly garbled output, copy-pasting was a pain)

AFAICT there appears to be some disagreement between the numpy, openmp and blas libraries, but I have no idea how to fix this.

Note that even on macOS and Linux I had to set numpy to >= 1.14 in host to get this working, as I got similar dependency issues. This time there appears to be no fix though.

I would be most grateful if you could give me some advice here! Please let me know if you need any additional information: I will gladly help out... spent most of my weekend doing this and I would really like to see this fixed 😄

isuruf commented 4 years ago

Can you give a link for the recipe?