search

conda-forge / gdal-feedstock

A conda-smithy repository for gdal.
BSD 3-Clause "New" or "Revised" License
30 stars 63 forks source link

Possible packing issue in python 2.7 #238

Closed scollis closed 5 years ago

scollis commented 5 years ago

Issue: Build done incorrectly (possibly) for GDAL against numpy-devel
Environment (conda list):

``` $ conda list arm_pyart 1.10.0 asn1crypto 0.24.0 py27_1003 conda-forge atomicwrites 1.2.1 py_0 conda-forge attrs 18.2.0 py_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.5 py_1 conda-forge backports_abc 0.5 py_1 conda-forge basemap 1.2.0 py27h50ae964_0 conda-forge blas 1.1 openblas conda-forge bzip2 1.0.6 1 conda-forge ca-certificates 2018.10.15 ha4d7672_0 conda-forge cartopy 0.17.0 py27h7547761_1 conda-forge certifi 2018.10.15 py27_1000 conda-forge cffi 1.11.5 py27h5e8e0c9_1 conda-forge cftime 1.0.2.1 py27h7eb728f_0 conda-forge chardet 3.0.4 py27_1003 conda-forge cryptography 2.3.1 py27hdffb7b8_0 conda-forge cryptography-vectors 2.3.1 py27_1000 conda-forge curl 7.62.0 h74213dd_0 conda-forge cvxopt 1.2.1 py27_blas_openblashfecbcd3_200 [blas_openblas] conda-forge cycler 0.10.0 py_1 conda-forge deprecation 2.0.5 py_0 conda-forge dsdp 5.8 blas_openblashddceaf2_202 [blas_openblas] conda-forge enum34 1.1.6 py27_1001 conda-forge fftw 3.3.8 h470a237_0 conda-forge freetype 2.9.1 h6debe1e_4 conda-forge funcsigs 1.0.2 py_2 conda-forge functools32 3.2.3.2 py_3 conda-forge futures 3.2.0 py27_1000 conda-forge gdal 1.10.1 py27_1 geos 3.6.2 hfc679d8_4 conda-forge glpk 4.65 h16a7912_1 conda-forge gmp 6.1.2 hfc679d8_0 conda-forge gsl 2.2.1 blas_openblashddceaf2_6 [blas_openblas] conda-forge h5py 2.8.0 py27h097b052_4 conda-forge hdf4 4.2.13 h951d187_2 conda-forge hdf5 1.10.3 hc401514_2 conda-forge icu 58.2 hfc679d8_0 conda-forge idna 2.7 py27_1002 conda-forge ipaddress 1.0.22 py_1 conda-forge jpeg 9c h470a237_1 conda-forge kiwisolver 1.0.1 py27h2d50403_2 conda-forge krb5 1.16.2 hbb41f41_0 conda-forge libcurl 7.62.0 hbdb9355_0 conda-forge libedit 3.1.20170329 haf1bffa_1 conda-forge libffi 3.2.1 hfc679d8_5 conda-forge libgfortran 3.0.0 1 conda-forge libiconv 1.15 h470a237_3 conda-forge libnetcdf 4.6.1 h350cafa_11 conda-forge libpng 1.6.35 ha92aebf_2 conda-forge libssh2 1.8.0 h5b517e9_3 conda-forge libtiff 4.0.9 he6b73bb_2 conda-forge libxml2 2.9.8 h422b904_5 conda-forge libxslt 1.1.32 h88dbc4e_2 conda-forge linecache2 1.0.0 py_1 conda-forge lxml 4.2.5 py27hc9114bc_0 conda-forge matplotlib 2.2.3 py27h0e0179f_0 conda-forge metis 5.1.0 hfc679d8_3 conda-forge more-itertools 4.3.0 py27_1000 conda-forge ncurses 6.1 hfc679d8_1 conda-forge netcdf4 1.4.2 py27hac939d9_0 conda-forge numpy 1.15.4 py27_blas_openblashb06ca3d_0 [blas_openblas] conda-forge olefile 0.46 py_0 conda-forge openblas 0.3.3 ha44fe06_1 conda-forge openssl 1.0.2p h470a237_1 conda-forge owslib 0.17.0 py_0 conda-forge packaging 18.0 py_0 conda-forge pathlib2 2.3.2 py27_1000 conda-forge pillow 5.3.0 py27hc736899_0 conda-forge pip 18.1 py27_1000 conda-forge pluggy 0.8.0 py_0 conda-forge proj4 5.2.0 h470a237_1 conda-forge py 1.7.0 py_0 conda-forge pycparser 2.19 py_0 conda-forge pyepsg 0.4.0 py_0 conda-forge pykdtree 1.3.1 py27h7eb728f_2 conda-forge pyopenssl 18.0.0 py27_1000 conda-forge pyparsing 2.3.0 py_0 conda-forge pyproj 1.9.5.1 py27h429999c_7 conda-forge pyshp 2.0.0 py_0 conda-forge pysocks 1.6.8 py27_1002 conda-forge pytest 4.0.1 py27_1000 conda-forge python 2.7.15 h33da82c_5 conda-forge python-dateutil 2.7.5 py_0 conda-forge pytz 2018.7 py_0 conda-forge readline 7.0 haf1bffa_1 conda-forge requests 2.20.1 py27_1000 conda-forge scandir 1.9.0 py27h470a237_0 conda-forge scipy 1.1.0 py27_blas_openblashb06ca3d_202 [blas_openblas] conda-forge setuptools 40.6.2 py27_0 conda-forge shapely 1.6.4 py27h164cb2d_1 conda-forge singledispatch 3.4.0.3 py27_1000 conda-forge six 1.11.0 py27_1001 conda-forge sqlite 3.25.3 hb1c47c0_0 conda-forge subprocess32 3.5.3 py27h470a237_0 conda-forge suitesparse 4.5.6 blas_openblash697e320_201 [blas_openblas] conda-forge tbb 2019.2 h2d50403_0 conda-forge tk 8.6.9 ha92aebf_0 conda-forge tornado 5.1.1 py27h470a237_0 conda-forge traceback2 1.4.0 py27_0 conda-forge trmm_rsl 1.49 2 conda-forge unittest2 1.1.0 py_0 conda-forge urllib3 1.23 py27_1001 conda-forge wheel 0.32.3 py27_0 conda-forge wradlib 1.2.0 py_0 conda-forge xmltodict 0.11.0 py_1 conda-forge xz 5.2.4 h470a237_1 conda-forge zlib 1.2.11 h470a237_3 conda-forge ```


Details about conda and system ( conda info ):

``` $ conda info active environment : testenv2 active env location : /Users/scollis/anaconda3/envs/testenv2 shell level : 1 user config file : /Users/scollis/.condarc populated config files : conda version : 4.5.11 conda-build version : 3.0.27 python version : 3.6.3.final.0 base environment : /Users/scollis/anaconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/osx-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/osx-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/osx-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/pro/osx-64 https://repo.anaconda.com/pkgs/pro/noarch package cache : /Users/scollis/anaconda3/pkgs /Users/scollis/.conda/pkgs envs directories : /Users/scollis/anaconda3/envs /Users/scollis/.conda/envs platform : osx-64 user-agent : conda/4.5.11 requests/2.18.4 CPython/3.6.3 Darwin/17.7.0 OSX/10.13.6 UID:GID : 80304931:20 netrc file : None offline mode : False ```

See https://github.com/OSGeo/gdal/issues/1120

Expected behavior and actual behavior.

Our toolkit uses GDAL for Geotiff generation. Our CI caught an issue testing against Python 2.7 on Travis CI. Note: it works with all Py3.x fine

https://travis-ci.org/ARM-DOE/pyart/jobs/460466036

Steps to reproduce the problem.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyart/io/output_to_geotiff.py:154: in write_grid_geotiff
    dst_ds.GetRasterBand(1).WriteArray(data[::-1, :])
../../../miniconda3/envs/testenv/lib/python2.7/site-packages/osgeo/gdal.py:1101: in WriteArray
    import gdalnumeric
../../../miniconda3/envs/testenv/lib/python2.7/site-packages/osgeo/gdalnumeric.py:1: in <module>
    from gdal_array import *
../../../miniconda3/envs/testenv/lib/python2.7/site-packages/osgeo/gdal_array.py:25: in <module>
    _gdal_array = swig_import_helper()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    def swig_import_helper():
        from os.path import dirname
        import imp
        fp = None
        try:
            fp, pathname, description = imp.find_module('_gdal_array', [dirname(__file__)])
        except ImportError:
>           import _gdal_array
E           ImportError: No module named _gdal_array

here is the miniconda env file: https://github.com/ARM-DOE/pyart/blob/master/continuous_integration/environment-2.7.yml

GDAL is not referenced specifically

test https://github.com/ARM-DOE/pyart/blob/master/pyart/io/tests/test_output_to_geotiff.py

code https://github.com/ARM-DOE/pyart/blob/master/pyart/io/output_to_geotiff.py

I need to track down where we are picking up gdal from

Operating system

Travis Linux

GDAL version and provenance

Tracking down

This issue is more to see if you are dropping 2.7 support... if this is the case we will no longer test our geotiff export for 2.7 we are dropping 2.7 soon anyway

scollis commented 5 years ago

It is likely that you have built the GDAL Python bindings without installing numpy-devel first.

kmuehlbauer commented 5 years ago

@scollis See my comment https://github.com/ARM-DOE/pyart/pull/787#issuecomment-442337807. This is most likely not a conda-forge issue, since numpy is pulled from defaults channel with an rather old version (1.11.3). Unfortunately I can't see the complete conda list of that testenv to figure out the provenience of gdal.

AFAIK conda-forge dropped creation of new Python2.7 packages and is based on Python3.6 and Python3.7 now. @ocefpaf can be more specific.

Please reopen if necessary.

ocefpaf commented 5 years ago

In your env list above your gdal is from defaults so this is likely a bad mix of packages. If you can afford to use only conda-forge, which most non-Windows users can, I'd recommend you to do that. If you are on Windows or need a package from defaults try to use only defaults, if you are on Windows and need package from both channels.... Sorry, no solution for it yet.

scollis commented 5 years ago

Perfect, thank you both. This is great information. I will try a conda-forge only env for testing. Failing this I will just turn off testing for GDAL deps in Py2.7. Thanks again, Scott

scollis commented 5 years ago

BTW just using conda-forge did not fix this.. but this is ok. I am now going to turn off our geotiff code for Py2.7

kmuehlbauer commented 5 years ago

@scollis If you add gdal to your environment-2.7.yml then an all conda-forge env is created.

scollis commented 5 years ago

Ooo! Ok. Good point.

From: Kai Mühlbauer notifications@github.com Reply-To: conda-forge/gdal-feedstock reply@reply.github.com Date: Wednesday, November 28, 2018 at 8:52 AM To: conda-forge/gdal-feedstock gdal-feedstock@noreply.github.com Cc: Scott scollis.acrf@gmail.com, Mention mention@noreply.github.com Subject: Re: [conda-forge/gdal-feedstock] Possible packing issue in python 2.7 (#238)

@scollis If you add gdal to your environment-2.7.yml then an all conda-forge env is created.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

scollis commented 5 years ago

Wow. That worked. Thanks @kmuehlbauer Next time we meet it will need to be at a brewery as I owe you many beers

ocefpaf commented 5 years ago

Wow. That worked. Thanks @kmuehlbauer

Note that if you remove defaults (or conda-forge) from your .condarc you will probably get rid of this issue for good (well... until you need something from both channels :smile:).

Once conda-forge finishes the compiler migration and we start better practices for pkgs variants, using both channels together will be a better experience than it is now.