PyPyType_Ready with tp_flags crashes PyPy interpreter

[gitlab-importer]

In Heptapod by bitbucket_importer on Feb 26, 2017, 06:03

Created originally on Bitbucket by denfromufa (Denis Akhiyarov)

I'm trying to debug PyPy crash with pythonnet library. PyPyType_Ready(type) works if no tp_flags are set on this type. If any basic flag is set (e.g. Py_TPFLAGS_DEFAULT_EXTERNAL) this crashes interpreter.

Are there any differences between CPython and PyPy wrt to setting tp_flags?

Here is a typical crash:

https://github.com/pythonnet/pythonnet/issues/330#issuecomment-280820452

>>>> import clr
Attempting to load 'Python.Runtime' using standard binding rules.
'Python.Runtime' not found using standard binding rules.
Attempting to load Python.Runtime from: 'C:\Python\pypy-c-jit-latest-win32\site-packages\Python.Runtime.dll'.
Success loading 'Python.Runtime' from: 'C:\Python\pypy-c-jit-latest-win32\site-packages\Python.Runtime.dll'.
RPython traceback:
  File "pypy_module_cpyext_2.c", line 25519, in type_realize
  File "pypy_module_cpyext_2.c", line 30900, in _type_realize
Fatal RPython error: AssertionError

[gitlab-importer]

In Heptapod by bitbucket_importer on Feb 26, 2017, 06:17

Created originally on Bitbucket by denfromufa (Denis Akhiyarov)

possibly related:

https://github.com/pybind/pybind11/blob/dd01665e5a8f5c13f18c009c34c1449c33b33dec/include/pybind11/class_support.h#L37

[gitlab-importer]

In Heptapod by bitbucket_importer on Mar 13, 2017, 21:45

Created originally on Bitbucket by mattip

Py_TPFLAGS_DEFAULT is actually a macro defined as Py_TPFLAGS_DEFAULT_EXTERNAL, and is used extensively in our tests. While there may be differences with the flags, and indeed some of the more obscure fields in the PyTypeObject are not yet properly handled, you will need to be more specific what type you are passing to PyType_Ready, Perhaps one of the many function structs are NULL or only partially filled in, or worse filled in by pythonnet after the call to PyType_Ready?

[gitlab-importer]

In Heptapod by bitbucket_importer on Mar 15, 2017, 05:38

Created originally on Bitbucket by mattip

Could you test again on a latest nightly? We (well, @arigo) has fixed a few issues around type creation over the past couple of days...

[gitlab-importer]

In Heptapod by bitbucket_importer on Mar 15, 2017, 06:01

Created originally on Bitbucket by denfromufa (Denis Akhiyarov)

@mattip thank you, let me try with latest PyPy 2.7 + cpyext towards the weekends. If the problem persists, then I plan to try with Mono on Linux. If nothing works, I plan to prepare a reproducible issue in a docker container.

Here is a typical hook from pythonnet to Python types, sorry you need to csharp :)

https://github.com/pythonnet/pythonnet/blob/master/src/runtime/typemanager.cs#L291

[gitlab-importer]

In Heptapod by @arigo on Mar 15, 2017, 07:23

Note that the request was to test with the latest nightly build (which we're about to turn into PyPy 5.7). Are you really using PyPy 2.7? That's a very old version for cpyext.

[gitlab-importer]

In Heptapod by @arigo on Mar 15, 2017, 07:28

Sorry! Confusion with version numbers. There was no PyPy version number 2.7; that's actually why we jumped from 2.6 to 4.0 in 2015. Of course you mean "PyPy2.7-v5.7".

[gitlab-importer]

In Heptapod by bitbucket_importer on Mar 18, 2017, 08:22

Created originally on Bitbucket by denfromufa (Denis Akhiyarov)

There is some improvement with latest nighly build. Particularly CreateMetaType method that I linked above now works as is.

But still the same crash with tp_flags set in at least one place:

internal static IntPtr CreateType(Type impl)

https://github.com/pythonnet/pythonnet/blob/master/src/runtime/typemanager.cs#L87

The only "hacks" that pythonnet does to tp_flags is re-using the integer slots #15 and #16 intended for stackless:

        // 15 and 16 are reserved for stackless
        public static int HaveStacklessExtension = 0;
        /* XXX Reusing reserved constants */
        public static int Managed = (1 << 15); // PythonNet specific
        public static int Subclass = (1 << 16); // PythonNet specific

[gitlab-importer]

In Heptapod by @arigo on Mar 18, 2017, 15:14

It helps if we get exact step-by-step information to reproduce the bug. Ideally, you could try to extract some C file that we can compile without any dependency. This would let us work with instructions like: "grab this foo.c and setup.py, run pypy setup.py foo.c, then run pypy -c "import foo", and it crashes like this for me (copy of the traceback, and version and OS information)."

[gitlab-importer]

In Heptapod by bitbucket_importer on Mar 18, 2017, 18:06

Created originally on Bitbucket by mattip

I doubt the crash is connected to tp_flags. Just to make sure, I tried out the value used in the initial issue report, tp_flags = 181227 and PyType_Ready works fine.

[gitlab-importer]

In Heptapod by @arigo on Mar 18, 2017, 20:05

@mattip: agreed. Anyway, all usages of tp_flags inside pypy check for one specific flag and ignore the rest.

[gitlab-importer]

In Heptapod by bitbucket_importer on Mar 19, 2017, 04:30

Created originally on Bitbucket by denfromufa (Denis Akhiyarov)

@mattip @arigo tp_flags = 181227 is not failing anymore on clrmetatype. But it fails when calling PyType_Ready on allocated type from this clrmetatype even when tp_flags is TypeFlags.Default=131563 or I guess anything:

https://github.com/pythonnet/pythonnet/blob/master/src/runtime/typemanager.cs#L74

I plan to work on reproducible example based on linux and mono next week, but the only language option is source C# and its compiled binaries. The compilation and building of extension is part of setup.py. But attached is the current state on Windows, both pypy and pythonnet packaged in this archive:

https://www.dropbox.com/s/946um8z9siicawh/pypy_17MAR2017.zip?dl=0

The build works like this:

pypy pythonnet\setup.py bdist_wheel
bin\pip install pythonnet\dist\pythonnet-2.2.2.tar.gz
mklink /H Python27.dll libpypy-c.dll
cd site-packages
mklink /H clr.pypy-41.pyd clr.pyd
pypy

import clr is enough to see the issue.

We had some progress this week.

[gitlab-importer]

In Heptapod by bitbucket_importer on Oct 26, 2017, 23:21

Created originally on Bitbucket by mattip

@denfromufa could you try with a nightly?

[gitlab-importer]

In Heptapod by bitbucket_importer on Nov 4, 2017, 04:54

Created originally on Bitbucket by denfromufa (Denis Akhiyarov)

@mattip what has changed?

[gitlab-importer]

In Heptapod by bitbucket_importer on Nov 4, 2017, 06:01

Created originally on Bitbucket by denfromufa (Denis Akhiyarov)

still the same crash:

#!python

Python 2.7.13 (d1cd247b10f6, Nov 04 2017, 00:36:53)
[PyPy 5.10.0-alpha0 with MSC v.1500 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``the zen attitude to programming:
reducing the oopses in your life''
>>>> import clr
Attempting to load 'Python.Runtime' using standard binding rules.
'Python.Runtime' not found using standard binding rules.
Attempting to load Python.Runtime from: 'C:\Python\pypy-c-jit-latest-win32-nov\site-packages\Python.Runtime.dll'.
Success loading 'Python.Runtime' from: 'C:\Python\pypy-c-jit-latest-win32-nov\site-packages\Python.Runtime.dll'.
RPython traceback:
  File "pypy_module_cpyext_2.c", line 11185, in type_realize
  File "pypy_module_cpyext_2.c", line 17149, in _type_realize
Fatal RPython error: AssertionError

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.