PyPy support

[xhochy]

It seems like we cannot cross-compile PyPy-based matrix entries currently. For example I come across with this in https://github.com/conda-forge/httptools-feedstock/pull/15

Is this a limitation of PyPy / just missing support here / an artifact that we force the python in build to be the same as the one in host?

[chrisburr]

I've just ran into this as well while changing to cross-compile ppc64le.

@isuruf Do you know if there is anything fundamental blocking this? Or is it just a matter of modifying the conda_build_config.yaml in this repo?

[xhochy]

crossenv doesn't seem to work out of the box with PyPy, so there probably is some work to be done.

[isuruf]

crossenv doesn't seem to work out of the box with PyPy, so there probably is some work to be done.

What's the error that you are getting?

[chrisburr]

I'm slowly working my way towards getting this working, one issue I'm running in to is that venv is broken:

$ mamba create --name test pypy^C
$ conda activate test
$ pypy -m venv /tmp/venv
Unable to symlink '/Users/cburr/mambaforge/envs/test/bin/../lib/python3.7' to '/tmp/venv/lib/python3.7'
Error: [Errno 17] File exists: '/tmp/venv/lib/python3.7'

I'll hack it for now so I can see if there are any other blockers.

@mattip I've seen a few recent mentions of this but only in the context of PyPy3.8. Is this also a known issue for PyPy3.7?

(For context: We'd like to use cross-compilation for the Python 3.10 migration but feedstocks with PyPy enabled prevent us from being able to)

[mattip]

I hadn't seen it in the context of the conda-forge pypy3.7 before, thanks. I opened conda-forge/pypy3.6-feedstock#58 to make sure it gets fixed for the next build of pypy

[chrisburr]

It seems the patches break venv creation on pypy3.7 since they change the layout to be like CPython

Aha! I wasn't expecting that and that makes a lot more sense 😄 Thanks for the quick response.

It would also be useful if VERSION and HOST_GNU_TYPE could be included in the build_time_vars dictionary in $PREFIX/lib_pypy/_sysconfigdata.py as crossenv makes use of it.

[mattip]

Sure. VERSION seems easy, it is the same as sysconfig.get_config_var('py_version_short)' (except on windows where it does not have the '.', so 39).

Could you add a table of values expected by crossenv to your docs, something like

key	example value	explanation
VERSION	3.9	%d.%d % sys.version_info[:2]
HOST_GNU_TPYE	x86_64-pc-linux-gnu	reported by running the <cpython>/config-guess shell script

I could help fill in the last column if you can provide the list of needed keys.

[chrisburr]

(I have no affiliation to crossenv, my main interest is in reducing the load on conda-forge's native aarch64 CI so we can start the cpython 3.10 migration, that said I'm happy to update the crossenv docs once I've got this working.)

I see that a lot of the issues I'm hitting are those discussed pypy#3483. The _sysconfigdata__linux_aarch64-linux-gnu.py file that will be included in PyPy v7.3.6 will make this much simplier and seems to include all of the important variables already. Is there a plan for when that will be released?

[chrisburr]

Using these two changes I've been able to cross-compile boost-histogram for PyPy:

• https://github.com/conda-forge/crossenv-feedstock/pull/17/files
• https://github.com/conda-forge/cross-python-feedstock/pull/29

Both PRs need some amount of cleanup though I'm not sure it's possibly to do cleanly before the next release of PyPy is made with:

• venv fixed
• the rendered _sysconfigdata__*.py files included in the build (so it's possible to statically extract the VERSION and SO_EXT)

I'm likely going to be offline for most of the next two weeks so if someone wants to take over and clean up these PRs up they're welcome to! 🙂

[mattip]

Is there a plan for when that will be released?

rc2 is out already but has a subtle bug in venv. I am adding HOST_GNU_TPYE and fixing venv for rc3, which should be out in a few days

[isuruf]

_PYTHON_SYSCONFIGDATA_NAME is still ignored. See the following comment from pypy sources.

def _init_posix(vars):
    """Initialize the module as appropriate for POSIX systems."""
    # CPython: _sysconfigdata_* is generated at build time, see _generate_posix_vars()
    # name = _get_sysconfigdata_name()
    # PyPy: _sysconfigdata is a dynamic module (uses sys to determine values),
    #       while _sysconfigdata_* is a static version of the same values
    #       generated by calling _generate_posix_vars as part of packaging.py.
    #       To prevent chicken-and-egg problems of needing the values before
    #       generating them, use the dynamic version here.
    name = '_sysconfigdata'
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
    build_time_vars = _temp.build_time_vars
    vars.update(build_time_vars)

@mattip, is there a env variable to figure out if we are building pypy or not?

[mattip]

If you are in the interpreter, you can use sys.implementation.name == 'pypy'.

If you are looking at the generated _sysconfigdata_* file from outside python (i.e. _sysconfigdata__linux_x86_64-linux-gnu.py), you can look at SO_ABI

[isuruf]

I mean, is it possible to figure out if we are building pypy or using an already built pypy?

[mattip]

I am not sure what you mean. Is this connected with _sysconfigdata?

[isuruf]

Yes, I found the following comment by you in pypy code.

PyPy: _sysconfigdata is a dynamic module (uses sys to determine values), while sysconfigdata* is a static version of the same values generated by calling _generate_posix_vars as part of packaging.py. To prevent chicken-and-egg problems of needing the values before generating them, use the dynamic version here.

Instead of using the dynamic version, the best option is to figure out if pypy is bootstrapping itself or not. Is there a way to figure out if pypy is bootstrapping itself?

[mattip]

Ahh that comment is at best confusing, and really is wrong. We should always use the dynamic values when inside the interpreter. The static values are useful for projects like crosscompile, that want to know things about an interpreter without actually running it. I should reformulate the comment, I wrote it before I had really thought about the implications. In hindsight, the dynamic values will always be correct, while the static values will be wrong if _generate_posix_vars is run at the wrong time (which currently is always the case: it gets the wrong path for INCLUDEPY).

[isuruf]

We should always use the dynamic values when inside the interpreter. The static values are useful for projects like crosscompile, that want to know things about an interpreter without actually running it.

That's fine for host, but for build, it needs to use _PYTHON_SYSCONFIGDATA_NAME env variable.

[isuruf]

How about something like below?

diff --git a/lib-python/3/sysconfig.py b/lib-python/3/sysconfig.py
index 0e3bf514a6..0e03f5088c 100644
--- a/lib-python/3/sysconfig.py
+++ b/lib-python/3/sysconfig.py
@@ -379,12 +379,7 @@ def get_makefile_filename():

 def _get_sysconfigdata_name():
-    return os.environ.get('_PYTHON_SYSCONFIGDATA_NAME',
-        '_sysconfigdata_{abi}_{platform}_{multiarch}'.format(
-        abi=sys.abiflags,
-        platform=sys.platform,
-        multiarch=getattr(sys.implementation, '_multiarch', ''),
-    ))
+    return os.environ.get('_PYTHON_SYSCONFIGDATA_NAME', '_sysconfigdata')

 def _generate_posix_vars(args):
@@ -460,14 +455,7 @@ def _generate_posix_vars(args):

 def _init_posix(vars):
     """Initialize the module as appropriate for POSIX systems."""
-    # CPython: _sysconfigdata_* is generated at build time, see _generate_posix_vars()
-    # name = _get_sysconfigdata_name()
-    # PyPy: _sysconfigdata is a dynamic module (uses sys to determine values),
-    #       while _sysconfigdata_* is a static version of the same values
-    #       generated by calling _generate_posix_vars as part of packaging.py.
-    #       To prevent chicken-and-egg problems of needing the values before
-    #       generating them, use the dynamic version here.
-    name = '_sysconfigdata'
+    name = _get_sysconfigdata_name()
     _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
     build_time_vars = _temp.build_time_vars
     vars.update(build_time_vars)

[mattip]

_get_sysconfigdata_name is used elsewhere. Perhaps

def _init_posix(vars):
     """Initialize the module as appropriate for POSIX systems."""
    name = os.environ.get('_PYTHON_SYSCONFIGDATA_NAME', '_sysconfigdata')
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
    build_time_vars = _temp.build_time_vars
    vars.update(build_time_vars)

[isuruf]

Yes, that should work.

[mattip]

I will add it to conda-forge/pypy3.6-feedstock#61

[mattip]

The fix was added in conda-forge/pypy3.6-feedstock#71. Was it sufficient?

[isuruf]

Yes, we can cross compile scipy pypy wheels now.

[isuruf]

Upstream PR at https://foss.heptapod.net/pypy/pypy/-/merge_requests/841