Patch: Support building C extensions for PyPy

[GoogleCodeExporter]

Using Shed Skin 0.7 and a PyPy nightly[1], SS fails to create a C extension due 
to a couple of issues:

 * infer.py calls gc.set_threshold
 * distutils.sysconfig fails to find config vars
 * when the issues above are fixed, generated C++ fails to build

The attached patch (against latest git) adds hacks for all issues mentioned 
above. Even with the patch, it's necessary to copy setobject.h from CPython 
into PyPy's include dir (I guess extensions using sets would be broken).

I'm running the test suite on 0.7 with CPython, 40 tests so far and all pass. 
Will report on results with PyPy (quite a lot) later. Running on tip seems 
broken before patch, though.

[1] http://buildbot.pypy.org/nightly/trunk/

Original issue reported on code.google.com by victorga...@gmail.com on 25 Jan 2011 at 6:36

Attachments:

• shedskinpypyext.diff

[GoogleCodeExporter]

thanks a lot for the patch! :) I committed it, with a few minor changes:

http://gitorious.org/shedskin/mainline/commit/562ae69ebb8e3be32ff1519aef47a3fd1f
1e51f7

- I don't think it hurts if we always add 'extern "C"' here
- I added a -p/--pypy option, since I think we want to be able to use pypy to 
generate a cpython-compatible extension module
- I didn't add the "setobject" include, because I don't think we want people to 
have to copy an include file over from CPython..? why doesn't PyPy have this?

please let me know if I screwed anything up. and thanks again! I'd be very 
interested in hearing how your further testing goes..

Original comment by mark.duf...@gmail.com on 26 Jan 2011 at 1:30

• Changed state: Accepted

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Sorry for not getting back sooner. I didn't try the git version yet, but it 
should be OK. Including setobjects.h will probably still be necessary to build, 
until PyPy has broader support for Python headers. I'll open an issue at PyPy's 
tracker for that.

An issue I forgot to mention is that PyPy nightlies only recognize .so files 
named according to PEP 3149[1], so I had to rename them .pypy-14.so for testing.

Below is a small report, including source from the attached files. It should be 
possible to run similar tests with Shed Skin's own examples compiled as 
extensions, and the results would surely be more meaningful. Even better if the 
example is also used in PyPy's speed tests[2][3][4].

[1]: http://www.python.org/dev/peps/pep-3149/
[2]: http://speed.pypy.org/changes/
[3]: http://codespeak.net/svn/pypy/benchmarks/own/
[4]: http://codespeak.net/svn/pypy/benchmarks/unladen_swallow/performance/

"""
A PyPy C++ extension generated by Shed Skin can show remarkable speed gains 
against the same code run as Python in PyPy. It's more than 12x faster for this 
example (30.45s PyPy vs 2.41s C extension). Of course, some code can get higher 
speedups and some code can get slower.

However, the call overhead for C code is huge in PyPy. For a trivial function, 
calling it a million times takes 0.74s in CPython vs 8.12s in PyPy.

Even in situations where overhead to speedup ratio makes it a net gain to use a 
C extension, care should be taken about silently changing semantics. Shed Skin 
doesn't support arbitrary-size arithmetic (Python longs) by default. Floating 
point code may also give different results due to implementation differences.
"""

# bench_overhead.py
#
# Extension code

def empty():
  return 1

def full():
  x = 0
  z = 0
  a = 0
  for b in xrange(1000):
    for y in xrange(100000):
      x += y / 10
      if x % 7 == 0:
        x = x / (b + 1) ** 2
      z += x / (b + 1)
    z /= 3
    a += (b + x) / ((y * b) + 1)
  return a, x, z

if __name__ == '__main__':
  empty()
  full()

# /Extension code
#
#########
#
# call_bench.py
#
# Benchmark code

import time
import sys
import bench_overhead
print "Benchmarking", sys.executable
print "Using", bench_overhead.__file__

empty = bench_overhead.empty
start = time.time()
for x in xrange(1000000):
  y = empty()
print time.time() - start

full = bench_overhead.full
start = time.time()
y = full()
print time.time() - start
print y

# /Benchmark code
#
#########
#
# Results
#
"""
# PyPy extension
Benchmarking /home/victor/bin/pypy-nightly
Using /home/victor/pypy-c-jit-41282/bench_overhead.pypy-14.so
8.12161111832
2.41368293762
(499950000, 49995, 985218)

# CPython extension
Benchmarking /usr/bin/python
Using /home/victor/pypy-c-jit-41282/bench_overhead.so
0.742517948151
2.90584301949
(499950000, 49995, 985218)

# PyPy pure Python
Benchmarking /home/victor/bin/pypy-nightly
Using /home/victor/pypy-c-jit-41282/bench_overhead.py
0.0694389343262
30.4507889748
(499950000, 49995, 985218L)
"""

Original comment by victorga...@gmail.com on 28 Jan 2011 at 3:24

Attachments:

• bench_overhead.py
• call_bench.py

[GoogleCodeExporter]

interesting! 

note that shedskin will probably become a lot slower if you pass/return 
arguments to 'empty', because builtin class instances are recursively 
copied/converted. no idea how pypy handles this. it might be interesting to 
also benchmark that a bit.. :-)

seeing the large difference for 'full', I think it would be useful if PyPy had 
(and perhaps it does) an option to also restrict integer semantics, because in 
many cases 64 or 32 bits are more than enough. it would be some work, but of 
course shedskin could also support arbitrary-size arithmetic (though libgmp or 
similar).

as long as the pypy devs stick to cpython semantics, I guess there is still a 
reason for me to develop shedskin.. :-)

Original comment by mark.duf...@gmail.com on 30 Jan 2011 at 12:29

[GoogleCodeExporter]

oh I shipped this option with 0.7.1, hope you don't mind.

Original comment by mark.duf...@gmail.com on 27 Feb 2011 at 12:41

[GoogleCodeExporter]

hm, do you feel I should keep this in for 0.8?

Original comment by mark.duf...@gmail.com on 25 May 2011 at 7:46

[GoogleCodeExporter]

Original comment by mark.duf...@gmail.com on 25 Jun 2011 at 8:13

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

I kept the code in 0.8, but the command-line option is now hidden.

Original comment by mark.duf...@gmail.com on 22 Jul 2011 at 6:59

• Changed state: Fixed