search

conda-forge / gdal-feedstock

A conda-smithy repository for gdal.
BSD 3-Clause "New" or "Revised" License
30 stars 63 forks source link

vsizip behaves strangely: Recode from CP437 to UTF-8 failed with the error: "Invalid argument". #83

Closed akorosov closed 7 years ago

akorosov commented 8 years ago

Problem After Installing gdal from conda-forge on fresh linux and attempting to open zip file with Sentinel data I got the following errors:

gdalinfo /vsizip//vagrant/shared/test_data/sentinel1a_l1/S1A_EW_GRDM_1SDH_20150702T172954_20150702T173054_006635_008DA5_55D1.zip/S1A_EW_GRDM_1SDH_20150702T172954_20150702T173054_006635_008DA5_55D1.SAFE/measurement/s1a-ew-grd-hv-20150702t172954-20150702t173054-006635-008da5-002.tiff
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
gdalinfo failed - unable to open '/vsizip//vagrant/shared/test_data/sentinel1a_l1/S1A_EW_GRDM_1SDH_20150702T172954_20150702T173054_006635_008DA5_55D1.zip/S1A_EW_GRDM_1SDH_20150702T172954_20150702T173054_006635_008DA5_55D1.SAFE/measurement/s1a-ew-grd-hv-20150702t172954-20150702t173054-006635-008da5-002.tiff'.

Workaround. Following the hint in this ticket I've set environment variable: export CPL_ZIP_ENCODING=UTF-8 Then it works as expected.

But why does it happen? It did not happen with gdal binaries from other channles (recipe)

Fix?

ocefpaf commented 8 years ago

@akorosov thanks for the report. I will investigate a fix for this. Can you provide a sample file with the problem so we can add that to our tests?

akorosov commented 8 years ago

stere.zip

gdalinfo /vsizip/stere.zip/stere.tif
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
gdalinfo failed - unable to open '/vsizip/stere.zip/stere.tif'.

After the workaround:

~>export CPL_ZIP_ENCODING=UTF-8
~>gdalinfo /vsizip/stere.zip/stere.tif
Driver: GTiff/GeoTIFF
Files: /vsizip/stere.zip/stere.tif
Size is 273, 265
Coordinate System is:
PROJCS["unnamed",
    GEOGCS["WGS 84",
        DATUM["unknown",
            SPHEROID["WGS84",6378137,298.257223563]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Stereographic"],
    PARAMETER["latitude_of_origin",71.4],
    PARAMETER["central_meridian",28.9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]
Origin = (-64455.359436834092776,72521.133364936860744)
Pixel Size = (500.000000000000000,-500.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  -64455.359,   72521.133) ( 27d 1'39.57"E, 72d 2'27.45"N)
Lower Left  (  -64455.359,  -59978.867) ( 27d 8'22.71"E, 70d51'14.56"N)
Upper Right (   72044.641,   72521.133) ( 30d59'33.46"E, 72d 2'19.42"N)
Lower Right (   72044.641,  -59978.867) ( 30d52' 2.96"E, 70d51' 7.03"N)
Center      (    3794.641,    6271.133) ( 29d 0'24.70"E, 71d27'22.22"N)
Band 1 Block=273x10 Type=Byte, ColorInterp=Red
  Metadata:
    dataType=1
    name=L_645
    ScaleRatio=0.203921568627
    standard_name=surface_upwelling_spectral_radiance_in_air_emerging_from_sea_water
Band 2 Block=273x10 Type=Byte, ColorInterp=Green
  Metadata:
    dataType=1
    name=L_555
    ScaleRatio=0.227450980392
    standard_name=surface_upwelling_spectral_radiance_in_air_emerging_from_sea_water
Band 3 Block=273x10 Type=Byte, ColorInterp=Blue
  Metadata:
    dataType=1
    name=L_469
    ScaleRatio=0.329411764706
    standard_name=surface_upwelling_spectral_radiance_in_air_emerging_from_sea_water
ocefpaf commented 8 years ago

Thanks! I will get to it soon.

ocefpaf commented 8 years ago

@akorosov I could not reprduce that error locally. Maybe we fixed it without knowing or maybe it is something specific to the system config. Either was I added a test case to catch that and if it re-surfaces in the CIs we will know.

Let me know that the new packages work for you.

ocefpaf commented 8 years ago

I was manually unpacking it and, of course, that does not really test vsizip. Looking into it again...

ocefpaf commented 8 years ago

@gillins I can confirm that conda-forge's vsizip does not work unless we set export CPL_ZIP_ENCODING=UTF-8.

The ticket mentions iconv, but I am not sure how that relates here.

Would it be OK to set that env var in the activation script?

gillins commented 8 years ago

Fine by me. Weird that is needed, but unlikely to conflict with anything else.

ocefpaf commented 8 years ago

Yeah. I am not sure what is going on there... I sent PR #105 where I added a proper test in the first commit, showing the failure on Linux, and the suggested fix. Since we cannot test the activation scripts at test time I had to hard-code the same env var in the tests scripts.

Can you review and merge #105?

ocefpaf commented 8 years ago

This is finally fixed in #105. Thanks @gillins and @akorosov!

akorosov commented 8 years ago

Seems that it comes back in gdal 2.1.1

ocefpaf commented 8 years ago

Can you do a conda list and send me the exact build number so I can check try to debug this?

akorosov commented 8 years ago

Hi @ocefpaf ! Here's the list:

vagrant@vagrant-ubuntu-trusty-64:~$ conda list
# packages in environment at /home/vagrant/miniconda:
#
cfunits                   1.1.4                     <pip>
backports.shutil_get_terminal_size 1.0.0                    py27_0    conda-forge
backports_abc             0.4                      py27_0    conda-forge
basemap                   1.0.8.dev0          np111py27_2    conda-forge
blas                      1.1                    openblas    conda-forge
cairo                     1.12.18                       8    conda-forge
certifi                   2016.8.2                 py27_0    conda-forge
conda                     4.1.11                   py27_1    conda-forge
conda-env                 2.5.2                    py27_0    conda-forge
configparser              3.5.0b2                  py27_0    conda-forge
curl                      7.49.1                        0    conda-forge
cycler                    0.10.0                   py27_0    conda-forge
decorator                 4.0.10                   py27_0    conda-forge
entrypoints               0.2.2                    py27_0    conda-forge
expat                     2.1.0                         1    conda-forge
fontconfig                2.11.1                        3    conda-forge
freetype                  2.6.3                         1    conda-forge
freexl                    1.0.2                         1    conda-forge
functools32               3.2.3.2                  py27_1    conda-forge
gdal                      2.1.1               np111py27_1    conda-forge
geos                      3.4.2                         2    conda-forge
gettext                   0.19.7                        1    conda-forge
giflib                    5.1.2                         1    conda-forge
glib                      2.48.0                        4    conda-forge
hdf4                      4.2.11                        4    conda-forge
hdf5                      1.8.17                        2    conda-forge
icu                       56.1                          4    conda-forge
ipykernel                 4.3.1                    py27_1    conda-forge
ipython                   5.0.0                    py27_1    conda-forge
ipywidgets                5.2.2                    py27_0    conda-forge
jinja2                    2.8                      py27_1    conda-forge
jpeg                      9b                            0    conda-forge
json-c                    0.12                          0    conda-forge
jsonschema                2.5.1                    py27_0    conda-forge
jupyter_client            4.3.0                    py27_0    conda-forge
jupyter_console           5.0.0                    py27_0    conda-forge
jupyter_core              4.1.0                    py27_0    conda-forge
kealib                    1.4.6                         3    conda-forge
libdap4                   3.18.0                        2    conda-forge
libffi                    3.2.1                         2    conda-forge
libiconv                  1.14                          3    conda-forge
libnetcdf                 4.4.1                         0    conda-forge
libpng                    1.6.23                        0    conda-forge
libsodium                 1.0.10                        0    conda-forge
libspatialite             4.3.0a                        9    conda-forge
libtiff                   4.0.6                         6    conda-forge
libuuid                   1.0.3                         1    conda-forge
libxml2                   2.9.4                         0    conda-forge
markupsafe                0.23                     py27_0    conda-forge
matplotlib                1.5.2               np111py27_6    conda-forge
mistune                   0.7.3                    py27_0    conda-forge
nbconvert                 4.2.0                    py27_0    conda-forge
nbformat                  4.0.1                    py27_0    conda-forge
netcdf4                   1.2.4               np111py27_2    conda-forge
nose                      1.3.7                    py27_0    conda-forge
notebook                  4.2.2                    py27_0    conda-forge
numpy                     1.11.1          py27_blas_openblas_201  [blas_openblas]  conda-forge
openblas                  0.2.18                        4    conda-forge
openjpeg                  2.1.0                         6    conda-forge
pango                     1.40.1                        0    conda-forge
pathlib2                  2.1.0                    py27_0    conda-forge
pcre                      8.38                          1    conda-forge
pexpect                   4.2.0                    py27_1    conda-forge
pickleshare               0.7.3                    py27_0    conda-forge
pillow                    3.3.0                    py27_0    conda-forge
pixman                    0.32.6                        0    conda-forge
postgresql                9.5.3                         2    conda-forge
proj.4                    4.9.2                         0    conda-forge
prompt_toolkit            1.0.4                    py27_0    conda-forge
ptyprocess                0.5.1                    py27_0    conda-forge
pygments                  2.1.3                    py27_0    conda-forge
pyparsing                 2.1.1                    py27_0    conda-forge
pyproj                    1.9.5.1                  py27_0    conda-forge
pyshp                     1.2.3                    py27_0    conda-forge
python-dateutil           2.5.3                    py27_0    conda-forge
pytz                      2016.6.1                 py27_0    conda-forge
pyzmq                     15.4.0                   py27_0    conda-forge
qt                        4.8.7                         0    conda-forge
qtconsole                 4.2.1                    py27_0    conda-forge
scipy                     0.18.0          np111py27_blas_openblas_201  [blas_openblas]  conda-forge
simplegeneric             0.8.1                    py27_0    conda-forge
singledispatch            3.4.0.3                  py27_0    conda-forge
sip                       4.18                     py27_0    conda-forge
six                       1.10.0                   py27_0    conda-forge
ssl_match_hostname        3.5.0.1                  py27_0    conda-forge
terminado                 0.6                      py27_0    conda-forge
tornado                   4.4.1                    py27_1    conda-forge
traitlets                 4.2.2                    py27_0    conda-forge
wcwidth                   0.1.7                    py27_0    conda-forge
widgetsnbextension        1.2.6                    py27_3    conda-forge
xerces-c                  3.1.4                         2    conda-forge
xz                        5.2.2                         0    conda-forge
zeromq                    4.1.5                         0    conda-forge
funcsigs                  1.0.2                     <pip>
harfbuzz                  0.9.39                        1  
ipython_genutils          0.1.0                    py27_0  
jupyter                   1.0.0                    py27_3  
libgfortran               3.0.0                         1  
mock                      2.0.0                     <pip>
openssl                   1.0.2h                        1  
pbr                       1.10.0                    <pip>
pip                       8.1.2                    py27_0  
pycosat                   0.6.1                    py27_1  
pycrypto                  2.6.1                    py27_0  
pyqt                      4.11.4                   py27_4  
python                    2.7.12                        1  
pyyaml                    3.11                     py27_4  
readline                  6.2                           2  
requests                  2.10.0                   py27_0  
ruamel_yaml               0.11.14                  py27_0  
setuptools                25.1.6                   py27_0  
sqlite                    3.13.0                        0  
tk                        8.5.18                        0  
wheel                     0.29.0                   py27_0  
yaml                      0.1.6                         0  
zlib                      1.2.8                         3

And here is the reproduced error:

vagrant@vagrant-ubuntu-trusty-64:~$ gdalinfo /vsizip/stere.zip/stere.tif
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
Warning 1: Recode from CP437 to UTF-8 failed with the error: "Invalid argument".
gdalinfo failed - unable to open '/vsizip/stere.zip/stere.tif'.

And the workaround fixes it:

vagrant@vagrant-ubuntu-trusty-64:~$ export CPL_ZIP_ENCODING=UTF-8
vagrant@vagrant-ubuntu-trusty-64:~$ gdalinfo /vsizip/stere.zip/stere.tif
Driver: GTiff/GeoTIFF
Files: /vsizip/stere.zip/stere.tif
Size is 273, 265
Coordinate System is:
PROJCS["unnamed",
    GEOGCS["WGS 84",
        DATUM["unknown",
            SPHEROID["WGS84",6378137,298.257223563]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Stereographic"],
    PARAMETER["latitude_of_origin",71.4],
    PARAMETER["central_meridian",28.9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]
Origin = (-64455.359436834092776,72521.133364936860744)
Pixel Size = (500.000000000000000,-500.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  -64455.359,   72521.133) ( 27d 1'39.57"E, 72d 2'27.45"N)
Lower Left  (  -64455.359,  -59978.867) ( 27d 8'22.71"E, 70d51'14.56"N)
Upper Right (   72044.641,   72521.133) ( 30d59'33.46"E, 72d 2'19.42"N)
Lower Right (   72044.641,  -59978.867) ( 30d52' 2.96"E, 70d51' 7.03"N)
Center      (    3794.641,    6271.133) ( 29d 0'24.70"E, 71d27'22.22"N)
Band 1 Block=273x10 Type=Byte, ColorInterp=Red
  Metadata:
    dataType=1
    name=L_645
    ScaleRatio=0.203921568627
    standard_name=surface_upwelling_spectral_radiance_in_air_emerging_from_sea_water
Band 2 Block=273x10 Type=Byte, ColorInterp=Green
  Metadata:
    dataType=1
    name=L_555
    ScaleRatio=0.227450980392
    standard_name=surface_upwelling_spectral_radiance_in_air_emerging_from_sea_water
Band 3 Block=273x10 Type=Byte, ColorInterp=Blue
  Metadata:
    dataType=1
    name=L_469
    ScaleRatio=0.329411764706
    standard_name=surface_upwelling_spectral_radiance_in_air_emerging_from_sea_water
vagrant@vagrant-ubuntu-trusty-64:~$
ocefpaf commented 8 years ago

Are you using it in the root env? If so conda does not export env vars. I checked it locally in a env and everything works.

PS: Please try echo $CPL_ZIP_ENCODING to see if the var exits before you defined it yourself.

akorosov commented 8 years ago

Yes, I'm using the root env and would prefer not to use a virtual environment, since we anyway use a virtual machine. CPL_ZIP_ENCODING does not exist before I define it myself.

Can that problem be fixed otherwise than setting an environmental variable? It was only a workaround that I found - not a solution. There are obviously some problems when compiling gdal that trigger this strange behavior. I never bumped into that before I switched to the conda-forge channel.

ocefpaf commented 8 years ago

Yes, I'm using the root env and would prefer not to use a virtual environment, since we anyway use a virtual machine. CPL_ZIP_ENCODING does not exist before I define it myself.

Be aware that conda won't execute the activation part of any software installed in the root env. So other things may be broken, like the data directory from gdal.

Can that problem be fixed otherwise than setting an environmental variable?

Not sure... If I find out I will certainly fix it! I will investigate this a little bit more. If you find anything let me know. Meanwhile lets keep this issue opened.

akorosov commented 8 years ago

Thank you, @ocefpaf ! I wish I could contribute but don't even understand where to start...

ocefpaf commented 8 years ago

I wish I could contribute but don't even understand where to start...

Same here. I am trying to single out libiconv to be certain, but so far nothing :unamused:

pkmn99 commented 7 years ago

I have exactly the same problem with gdal 2.1.3. The export trick can fix it. Don't know why it happens. There is no such problem on another machine

ocefpaf commented 7 years ago

@akorosov I am fixing this as a "won't fix" b/c we never found out the root of the issue and we do have a workaround for it.

WeatherGod commented 6 years ago

@ocefpaf A little bit of a clarification is needed, as I just ran into this problem as I was setting up an automated CI for a package of mine that uses gdal. It isn't that conda doesn't execute the activation part of any software in the root environment -- it does (or at least, it does now). The issue is that the activation isn't performed as part of the install. I was trying to figure out why I was getting these failures for a CI testing that looked like:

conda create -n foobar python=3.6
source activate foobar
conda install -c conda-forge --file conda-requirements.txt
pip install -e .

Did you see the mistake? I made a non root environment and activated it and then installed packages like gdal. So the gdal activation never happened! Instead, I should have done the following:

conda create -n foobar -c conda-forge --file conda-requirements.txt python=3.6
source activate foobar
pip install -e .
ocefpaf commented 6 years ago

It isn't that conda doesn't execute the activation part of any software in the root environment -- it does (or at least, it does now).

Cool. I knew that was in the works but did not know it already landed.

Yep. I actually made that mistake a few times in the past and failed to document it properly. I believe conda can "re-activate" an env in case you are installing more packages and testing things interactively. Although for a CI it makes sense to just flip it like you did. Thanks!