ValueError: array.array size changed, may indicate binary incompatibility on PyPy

[cgohlke]

Importing pyproj 3.0.1 on PyPy for Windows raises the following error:

Python 3.7.10 (77787b8f4c49, May 15 2021, 11:51:36)
[PyPy 7.3.5 with MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>> import pyproj
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "X:\pypy3\site-packages\pyproj\__init__.py", line 64, in <module>
    from pyproj.proj import Proj, pj_list  # noqa: F401
  File "X:\pypy3\site-packages\pyproj\proj.py", line 21, in <module>
    from pyproj._transformer import Factors, _Transformer
  File "pyproj\_transformer.pyx", line 1, in init pyproj._transformer
ValueError: array.array size changed, may indicate binary incompatibility. Expected 72 from C header, got 24 from PyObject

The issue is that there is "no clean C interface to the array.array object in Python". See discussion at https://github.com/h5py/h5py/issues/1514 and https://github.com/h5py/h5py/pull/1515 for a possible fix.

[snowman2]

Related: https://github.com/conda-forge/pyproj-feedstock/pull/90

[stonebig]

hope ?

[snowman2]

This may be useful: https://github.com/python/cpython/blob/a045991f60b51636a784623dda7ad84b5b2c6b73/Include/tupleobject.h#L30

[snowman2]

For reference: https://foss.heptapod.net/pypy/pypy/-/issues/3039 (from: https://github.com/conda-forge/pyproj-feedstock/pull/90#issuecomment-795674854)

[stonebig]

I'm lost in thoughts, it seems the problem is back with PyPy-3.7-7.3.6, and now Pyproj is droping support for 3.7. I may give up and hope for a better PyPy-3.8.

# plotnine: giving a taste of ggplot of R langage (formerly we were using ggpy) from plotnine import ggplot, aes, geom_blank, geom_point, stat_smooth, facet_wrap, theme_bw from plotnine.data import mtcars ggplot(mtcars, aes(x='hp', y='wt', color='mpg')) + geom_point() +\ facet_wrap("~cyl") + theme_bw() --------------------------------------------------------------------------- ValueError Traceback (most recent call last) C:\Users\helene\AppData\Local\Temp/ipykernel_2480/2389138475.py in 1 # plotnine: giving a taste of ggplot of R langage (formerly we were using ggpy) ----> 2 from plotnine import ggplot, aes, geom_blank, geom_point, stat_smooth, facet_wrap, theme_bw 3 from plotnine.data import mtcars 4 ggplot(mtcars, aes(x='hp', y='wt', color='mpg')) + geom_point() +\ 5 facet_wrap("~cyl") + theme_bw() C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\__init__.py in ----> 1 from .qplot import qplot # noqa: F401 2 from .ggplot import ggplot, ggsave # noqa: F401 3 from .ggplot import save_as_pdf_pages # noqa: F401 4 from .watermark import watermark # noqa: F401 5 from .mapping import * # noqa: F401,F403,E261 C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\qplot.py in 7 from patsy.eval import EvalEnvironment 8 ----> 9 from .ggplot import ggplot 10 from .mapping.aes import aes, all_aesthetics, scaled_aesthetics 11 from .labels import labs C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\ggplot.py in 18 from .mapping.aes import aes, make_labels 19 from .layer import Layers ---> 20 from .facets import facet_null 21 from .facets.layout import Layout 22 from .options import get_option C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\facets\__init__.py in ----> 1 from .facet_grid import facet_grid 2 from .facet_null import facet_null 3 from .facet_wrap import facet_wrap 4 from .labelling import label_value, label_both, label_context 5 from .labelling import labeller, as_labeller C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\facets\facet_grid.py in 4 from ..utils import match, join_keys 5 from ..exceptions import PlotnineError ----> 6 from .facet import facet, layout_null, combine_vars, add_missing_facets 7 from .facet import eval_facet_vars 8 from .strips import strip C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\facets\facet.py in 11 from ..utils import cross_join, match 12 from ..exceptions import PlotnineError, PlotnineWarning ---> 13 from ..scales.scales import Scales 14 15 # For default matplotlib backend C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\scales\__init__.py in 81 from .scale_xy import scale_y_timedelta 82 # limits ---> 83 from .limits import xlim, ylim, lims, expand_limits 84 85 C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\scales\limits.py in 6 7 from ..mapping.aes import aes, all_aesthetics ----> 8 from ..geoms import geom_blank 9 from ..scales.scales import make_scale 10 from ..exceptions import PlotnineError C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\geoms\__init__.py in 25 from .geom_line import geom_line 26 from .geom_linerange import geom_linerange ---> 27 from .geom_map import geom_map 28 from .geom_path import geom_path 29 from .geom_point import geom_point C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\plotnine\geoms\geom_map.py in 5 6 try: ----> 7 import geopandas # noqa: F401 8 except ImportError: 9 HAS_GEOPANDAS = False C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\geopandas\__init__.py in ----> 1 from geopandas._config import options # noqa 2 3 from geopandas.geoseries import GeoSeries # noqa 4 from geopandas.geodataframe import GeoDataFrame # noqa 5 from geopandas.array import points_from_xy # noqa C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\geopandas\_config.py in 107 use_pygeos = Option( 108 key="use_pygeos", --> 109 default_value=_default_use_pygeos(), 110 doc=( 111 "Whether to use PyGEOS to speed up spatial operations. The default is True " C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\geopandas\_config.py in _default_use_pygeos() 93 94 def _default_use_pygeos(): ---> 95 import geopandas._compat as compat 96 97 return compat.USE_PYGEOS C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\geopandas\_compat.py in 7 import numpy as np 8 import pandas as pd ----> 9 import pyproj 10 import shapely 11 import shapely.geos C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\pyproj\__init__.py in 47 import warnings 48 ---> 49 import pyproj.network 50 from pyproj._datadir import ( # noqa: F401 pylint: disable=unused-import 51 _pyproj_global_context_initialize, C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.6-win64\site-packages\pyproj\network.py in 8 import certifi 9 ---> 10 from pyproj._network import ( # noqa: F401 pylint: disable=unused-import 11 _set_ca_bundle_path, 12 is_network_enabled, pyproj\_network.pyx in init pyproj._network() ValueError: array.array size changed, may indicate binary incompatibility. Expected 72 from C header, got 24 from PyObject

[snowman2]

@stonebig, are you using the latest pyproj version?

[stonebig]

pyproj-3.2.1

I have lot of (new ?) problems of messageless crash with PyPy-3.7-7.3.6 just released (and rc3, and rc2). But this one was supposed to be fixed. So I hope it's an hint to all the other problems.

[cgohlke]

I have lot of (new ?) problems of messageless crash with PyPy-3.7-7.3.6 just released (and rc3, and rc2).

That is also my experience. I get reproducible crashes in numpy's _multiarray_umath extension. Unfortunately pypy does not ship with debug symbols. The PyErr_SetFromWindowsErr function seems to have changed ABI. Not sure that is causing these problems. I'll skip PyPy-3.7-7.3.6 for now. Recompiling is not an option.

[stonebig]

there seems to be maybe a hint from https://libera.irclog.whitequark.org/pypy/2021-10-18 "more fallout from the release: it seems CPython on windows moved to linking with libffi-8 when we link with libffi-7"

[mattip]

@cgohlke PyPy-3.7-7.3.6 broke binary compatibility with previous versions. I released 7.3.7 to revert the breaking changes. Sorry for the mess.

[cgohlke]

Thank you! Do you think it is safe to start building for pypy3.8-v7.3.7?

[mattip]

I think so. it would be nice to get some feedback. Issues of failures can be opened on the PyPy bug tracker

[stonebig]

Hi. I'm a bit lost in thoughts,

On My PC, with pypy3.7-v7.3.7-win64.zip and re-downloaded pyproj-3.2.1-pp37-pypy37_pp73-win_amd64.whl wheel , this basic test from first cells of https://github.com/pyproj4/pyproj/blob/master/docs/advanced_examples.rst fails:

import numpy as np
from pyproj import Transformer, transform

transformer = Transformer.from_crs(2263, 4326)
x_coords = np.random.randint(80000, 120000)
y_coords = np.random.randint(200000, 250000)

Python 3.7.12 (44db26267d0a, Oct 24 2021, 14:22:11)
[PyPy 7.3.7 with MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> 
========== RESTART: C:/WinP/bd37/buPyPy/WPy64-37120/notebooks/test.py ==========
Traceback (most recent call last):
  File "C:/WinP/bd37/buPyPy/WPy64-37120/notebooks/test.py", line 2, in <module>
    from pyproj import Transformer, transform
  File "C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.7-win64\site-packages\pyproj\__init__.py", line 49, in <module>
    import pyproj.network
  File "C:\WinP\bd37\buPyPy\WPy64-37120\pypy3.7-v7.3.7-win64\site-packages\pyproj\network.py", line 10, in <module>
    from pyproj._network import (  # noqa: F401 pylint: disable=unused-import
  File "pyproj\_network.pyx", line 1, in init pyproj._network
ValueError: array.array size changed, may indicate binary incompatibility. Expected 72 from C header, got 24 from PyObject
>>> 

[mattip]

The only binary wheel for PyPy on https://pypi.org/project/pyproj/#files is one for windows. Perhaps somehow it missed the change in #883?

[snowman2]

@cgohlke creates the Windows wheels. The wheels for Linux & OSX are built here: https://github.com/pyproj4/pyproj-wheels/. Contributions welcome =).

[stonebig]

@cgohlke made another wheel, so problem moved down to rtree wheel now, that seems a bit lost with pypy3.7-v7.3.7 old way of things

pypy3.8-v7.3.7 is clother to cPython standard, so unmaintained rtree may feel better.

see also rtreee issue https://github.com/Toblerity/rtree/issues/191

[stonebig]

after copying around rtree DLLs, another tree hit

Python 3.7.12 (44db26267d0a, Oct 24 2021, 14:22:11)
[PyPy 7.3.7 with MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> 
=========== RESTART: C:\WinP\bd37\buPyPy\WPy64-37120\notebooks\sea.py ==========

================================ RESTART: Shell ================================
>>> 
========= RESTART: C:/WinP/bd37/buPyPy/WPy64-37120/notebooks/pyproj.py =========
Traceback (most recent call last):
  File "C:/WinP/bd37/buPyPy/WPy64-37120/notebooks/pyproj.py", line 2, in <module>
    from pyproj import Transformer, transform
  File "C:/WinP/bd37/buPyPy/WPy64-37120/notebooks\pyproj.py", line 2, in <module>
    from pyproj import Transformer, transform
ImportError: cannot import name 'Transformer'
>>> 

[stonebig]

transformer.py seems lowercase on Windows ... playing lowercase with the example or the 'transformer.py' leads me to :

    transformer = transformer.from_crs(2263, 4326)
AttributeError: module 'pyproj.transformer' has no attribute 'from_crs'

[stonebig]

Back to this problem on PyPy-3.8rc1+nightly.... ... it's the only binary problem I still detect, as PyPy-3.8 has now roughly as much compatible packages as PyPy-3.7

[mattip]

@stonebig could you confirm you are using pyproj 3.3 from https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyproj, and that this line fails?

>>>> import pyproj

[cgohlke]

I have once again recompiled the pyproj package for PyPy and it is importing again. I think this issue is due to Cython producing incompatible versions of C files when cythonizing on CPython vs PyPy, which makes it hard and error prone to build CPython and PyPy wheels from the same copy of the source without removing the cythonized files in between builds.

[mattip]

Thanks @cgohlke. It seems this cython-translation time directive, which was added in #883, means that the cython c-code will indeed change depending on implementation. Over at cython/cython#3448, the discussion of this problem ended with

I think it's ok to consider the array.array internals CPython specific. Using them is not (currently) compatible with PyPy. The decision is then up to the users.

and cython HEAD emits a warning when using array.array on PyPy.

In h5py a similar problem was solved by replacing a local variable that used array.array('B', n) with <char *>emalloc(n), but it seems replacing ~create_array~ empty_array with something that requires a deallocation step will be more intrusive for pyproj, since the returned named tuple from GeodIntermediateReturn has array.array for lons, lats, and azis (sp?).

Perhaps the translation time directive could be replaced with a runtime check for sys.implementation.name == 'pypy'. This would work because the cython code from cpython cimport array; cdef array.array compiles on PyPy but would not run at runtime so the warning would not be emitted (the alternative code path is slightly slower).