search

conda-forge / rasterio-feedstock

A conda-smithy repository for rasterio.
BSD 3-Clause "New" or "Revised" License
42 stars 22 forks source link

Segmentation fault when using Resampling.max method #163

Closed sebhan2 closed 4 years ago

sebhan2 commented 4 years ago

I want to resample a georeferend raster .tif file with Python using rasterio. Here is the example I found example and the documentaion of the resampling methods.

When I am using the following methods ‘max’, ‘min’, ‘med’, ‘q1’, ‘q3’ which are only supported in GDAL >= 2.0.0 the program fails.

I am using a conda environment and rasterio.gdal_version gives me version 3.0.4.

When I run the following code (i.e. almost the example code):

from pathlib import Path
raster = Path('dsm_EPSG_4839_cropped.tif')

import rasterio
from rasterio.enums import Resampling

upscale_factor = 1/(2**5)

with rasterio.open(raster) as dataset:

    # resample data to target shape
    data = dataset.read(
        out_shape=(
            dataset.count,
            int(dataset.height * upscale_factor),
            int(dataset.width * upscale_factor)
        ),
        resampling=Resampling.max
    )

    # scale image transform
    transform = dataset.transform * dataset.transform.scale(
        (dataset.width / data.shape[-1]),
        (dataset.height / data.shape[-2])
    )

I am getting the following error:

Segmentation fault: 11

How do I solve the issue and get the example to run without the segmentation fault?

conda list:
#
# Name                    Version                   Build  Channel
affine                    2.3.0                      py_0    conda-forge
appnope                   0.1.0           py37hc8dfbb8_1001    conda-forge
attrs                     19.3.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
bleach                    3.1.5              pyh9f0ad1d_0    conda-forge
bokeh                     2.0.2                    py37_0  
boost-cpp                 1.72.0               hdf9ef73_0    conda-forge
brotlipy                  0.7.0           py37h9bfed18_1000    conda-forge
bzip2                     1.0.8                h0b31af3_2    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
cairo                     1.16.0            hec6a9b0_1003    conda-forge
cartopy                   0.18.0           py37h08e9697_0    conda-forge
certifi                   2020.4.5.1       py37hc8dfbb8_0    conda-forge
cffi                      1.14.0           py37h356ff06_0    conda-forge
cfitsio                   3.470                hddc9c06_5    conda-forge
cftime                    1.1.3            py37h10e2902_0    conda-forge
chardet                   3.0.4           py37hc8dfbb8_1006    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                      py_0    conda-forge
cloudpickle               1.4.1                      py_0    conda-forge
colorcet                  2.0.2                      py_0  
cryptography              2.9.2            py37he655712_0    conda-forge
curl                      7.69.1               h2d98d24_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cytoolz                   0.10.1           py37h0b31af3_0    conda-forge
dask                      2.16.0                     py_0    conda-forge
dask-core                 2.16.0                     py_0    conda-forge
datashader                0.10.0                     py_0    conda-forge
datashape                 0.5.4                      py_1    conda-forge
decorator                 4.4.2                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
distributed               2.16.0           py37hc8dfbb8_0    conda-forge
entrypoints               0.3             py37hc8dfbb8_1001    conda-forge
expat                     2.2.9                h4a8c4bd_2    conda-forge
fiona                     1.8.13           py37h9c7205d_1    conda-forge
fontconfig                2.13.1            h6b1039f_1001    conda-forge
freetype                  2.10.2               h8da9a1a_0    conda-forge
freexl                    1.0.5             h1de35cc_1002    conda-forge
fsspec                    0.7.3                      py_0    conda-forge
gdal                      3.0.4           py37h08e9697_10    conda-forge
geopandas                 0.7.0                      py_1    conda-forge
geos                      3.8.1                h4a8c4bd_0    conda-forge
geotiff                   1.6.0                hd8796ba_0    conda-forge
geoviews                  1.8.1                      py_0    conda-forge
geoviews-core             1.8.1                      py_0    conda-forge
gettext                   0.19.8.1          h46ab8bc_1002    conda-forge
giflib                    5.2.1                h0b31af3_2    conda-forge
glib                      2.64.2               h577aef8_1    conda-forge
hdf4                      4.2.13            h84186c3_1003    conda-forge
hdf5                      1.10.6          nompi_h3e39495_100    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
holoviews                 1.13.2             pyh9f0ad1d_0    conda-forge
icu                       64.2                 h6de7cb9_1    conda-forge
idna                      2.9                        py_1    conda-forge
imagecodecs-lite          2019.12.3        py37h9bfed18_0    conda-forge
imageio                   2.8.0                      py_0    conda-forge
importlib-metadata        1.6.0            py37hc8dfbb8_0    conda-forge
importlib_metadata        1.6.0                         0    conda-forge
ipykernel                 5.2.1            py37h43977f1_0    conda-forge
ipython                   7.14.0           py37hc8dfbb8_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.17.0           py37hc8dfbb8_0    conda-forge
jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
jpeg                      9c                h1de35cc_1001    conda-forge
json-c                    0.13.1            h575e443_1002    conda-forge
jsonschema                3.2.0            py37hc8dfbb8_1    conda-forge
jupyter_client            6.1.3                      py_0    conda-forge
jupyter_core              4.6.3            py37hc8dfbb8_1    conda-forge
kealib                    1.4.13               h40102fb_1    conda-forge
kiwisolver                1.2.0            py37ha1cc60f_0    conda-forge
krb5                      1.17.1               h1752a42_0    conda-forge
libblas                   3.8.0               16_openblas    conda-forge
libcblas                  3.8.0               16_openblas    conda-forge
libcurl                   7.69.1               hc0b9707_0    conda-forge
libcxx                    10.0.0               h1af66ff_2    conda-forge
libdap4                   3.20.6               h8c15375_0    conda-forge
libedit                   3.1.20181209         hb402a30_0  
libffi                    3.2.1             h4a8c4bd_1007    conda-forge
libgdal                   3.0.4               h242383b_10    conda-forge
libgfortran               4.0.0                         2    conda-forge
libiconv                  1.15              h0b31af3_1006    conda-forge
libkml                    1.3.0             h169b8f9_1011    conda-forge
liblapack                 3.8.0               16_openblas    conda-forge
libnetcdf                 4.7.4           nompi_ha11d67f_104    conda-forge
libopenblas               0.3.9                h3d69b6c_0    conda-forge
libpng                    1.6.37               hbbe82c9_1    conda-forge
libpq                     12.2                 h489d428_1    conda-forge
libsodium                 1.0.17               h01d97ff_0    conda-forge
libspatialindex           1.9.3                h4a8c4bd_3    conda-forge
libspatialite             4.3.0a            h658e6c1_1038    conda-forge
libssh2                   1.9.0                ha12b0ac_1  
libtiff                   4.1.0                h2ae36a8_6    conda-forge
libwebp-base              1.1.0                h0b31af3_3    conda-forge
libxml2                   2.9.10               h53d96d6_0    conda-forge
llvm-openmp               10.0.0               h28b9765_0    conda-forge
llvmlite                  0.32.1           py37h8c7ce04_0  
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.9.2                h4a8c4bd_1    conda-forge
markdown                  3.2.2                      py_0    conda-forge
markupsafe                1.1.1            py37h9bfed18_1    conda-forge
matplotlib-base           3.2.1            py37hddda452_0    conda-forge
mistune                   0.8.4           py37h9bfed18_1001    conda-forge
msgpack-python            1.0.0            py37ha1cc60f_1    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
munch                     2.5.0                      py_0    conda-forge
nbconvert                 5.6.1            py37hc8dfbb8_1    conda-forge
nbformat                  5.0.6                      py_0    conda-forge
ncurses                   6.2                  h0a44026_1  
netcdf4                   1.5.3           nompi_py37hf55ae24_105    conda-forge
networkx                  2.4                        py_1    conda-forge
notebook                  6.0.3            py37hc8dfbb8_0    conda-forge
numba                     0.49.1           py37h86efe34_0  
numpy                     1.18.4           py37h7687784_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openjpeg                  2.3.1                h254dc36_3    conda-forge
openssl                   1.1.1                h1de35cc_0    anaconda
owslib                    0.19.2                     py_1    conda-forge
packaging                 20.3                       py_0  
pandas                    1.0.3            py37h94625e5_1    conda-forge
pandoc                    2.9.2.1                       0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
panel                     0.9.5                      py_1    conda-forge
param                     1.9.3                      py_0    conda-forge
parso                     0.7.0              pyh9f0ad1d_0    conda-forge
partd                     1.1.0                      py_0    conda-forge
pcre                      8.44                 h4a8c4bd_0    conda-forge
pexpect                   4.8.0            py37hc8dfbb8_1    conda-forge
pickleshare               0.7.5           py37hc8dfbb8_1001    conda-forge
pillow                    7.1.2            py37hfd78ece_0    conda-forge
pip                       20.1               pyh9f0ad1d_0    conda-forge
pixman                    0.38.0            h01d97ff_1003    conda-forge
poppler                   0.87.0               h3232a60_1    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                12.2                 h62ab893_1    conda-forge
proj                      7.0.0                hbce2d5c_4    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt-toolkit            3.0.5                      py_0    conda-forge
psutil                    5.7.0            py37h9bfed18_1    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pycparser                 2.20                       py_0    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pygments                  2.6.1                      py_0    conda-forge
pyopenssl                 19.1.0                     py_1    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    2.6.1.post1      py37hbd4ead9_0    conda-forge
pyrsistent                0.16.0           py37h9bfed18_0    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.1            py37hc8dfbb8_1    conda-forge
python                    3.7.7           hc70fcce_0_cpython  
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2020.1             pyh9f0ad1d_0    conda-forge
pyviz_comms               0.7.4              pyh8c360ce_0    conda-forge
pywavelets                1.1.1            py37h10e2902_1    conda-forge
pyyaml                    5.3.1            py37h9bfed18_0    conda-forge
pyzmq                     19.0.1           py37haec44b1_0    conda-forge
rasterio                  1.1.3            py37he71f6a4_0    conda-forge
readline                  8.0                  h1de35cc_0  
requests                  2.23.0             pyh8c360ce_2    conda-forge
rtree                     0.9.4            py37h8526d28_1    conda-forge
scikit-image              0.17.2           py37h94625e5_0    conda-forge
scipy                     1.4.1            py37hce1b9e5_3    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                46.3.1           py37hc8dfbb8_0    conda-forge
shapely                   1.7.0            py37hfcf0db4_3    conda-forge
six                       1.14.0                     py_1    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
sqlite                    3.31.1               h5c1f38d_1  
tbb                       2018.0.5             h2d50403_0    conda-forge
tblib                     1.6.0                      py_0    conda-forge
terminado                 0.8.3            py37hc8dfbb8_1    conda-forge
testpath                  0.4.4                      py_0    conda-forge
tifffile                  2020.5.11                  py_0    conda-forge
tiledb                    1.7.7                h84aa2a7_1    conda-forge
tk                        8.6.10               hbbe82c9_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.4            py37h9bfed18_1    conda-forge
tqdm                      4.46.0             pyh9f0ad1d_0    conda-forge
traitlets                 4.3.3            py37hc8dfbb8_1    conda-forge
typing_extensions         3.7.4.2                    py_0    conda-forge
tzcode                    2020a                h0b31af3_0    conda-forge
urllib3                   1.25.9                     py_0    conda-forge
wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.34.2                     py_1    conda-forge
xarray                    0.15.1                     py_0    conda-forge
xerces-c                  3.2.2             h8f8adb3_1004    conda-forge
xz                        5.2.5                h0b31af3_0    conda-forge
yaml                      0.2.4                h0b31af3_0    conda-forge
zeromq                    4.3.2                h6de7cb9_2    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge
zstd                      1.4.4                h4b3e974_3    conda-forge
ocefpaf commented 4 years ago

I don't have your file to test but I could not reproduce the problem with a simple test with the same example.tif from rasterio docs. Closing this but I can test again with your file if you can provide it.

sebhan2 commented 4 years ago

the file is a large raster tiff of approximately 2.5 GB and the file is working with other methoads like average, nearest neighbor etc. Is there a limit on the input file size? Do you have an idea were to look to find such an error?

ocefpaf commented 4 years ago

the file is a large raster tiff of approximately 2.5 GB

OK. We are not sharing that 😄

and the file is working with other methods like average, nearest neighbor etc.

Also, ignore what I mentioned above. It crashes with:

ERROR 1: GDALGetResampleFunction: Unsupported resampling method "UNKNOWN".

I ran it as a script before and I thought it completed but it hid the error message.

snowman2 commented 4 years ago

Does this happen with the rasterio wheels?

sebhan2 commented 4 years ago

rasterio wheels is the version installed via pip with the command pip install rasterio, correct? If that is the case then no it does not.

ocefpaf commented 4 years ago

Our gdal build is quite different from the one used to build wheels, it would be nice to find out what is causing this but unfortunately I do not have the cycles at the moment to investigate this.

JesseCrocker commented 4 years ago

Related issue on rasterio repo https://github.com/mapbox/rasterio/issues/1930 Related issue on gdal repo https://github.com/OSGeo/gdal/issues/2587

ocefpaf commented 4 years ago

rasterio wheels is the version installed via pip with the command pip install rasterio, correct? If that is the case then no it does not.

I could reproduce the failure with the Wheels and thanks to @JesseCrocker it is already reported upstream. Also, it seems that a fix is already on the way. Closing this b/c it is not a problem with the conda package.