search

alanjds / grumpy

**MOVED TO NEW MOTHER ORG `grumpyhome/grumpy`** Grumpy is a Python to Go source code transcompiler and runtime.
https://github.com/grumpyhome/grumpy
Apache License 2.0
16 stars 1 forks source link

Use PyPy for pure python code #115

Open alanjds opened 6 years ago

alanjds commented 6 years ago

On the current Readme it states: "But there are also a number of libraries in CPython that are C extension modules that need to be rewritten. This includes re, itertools and others."

I think that at least itertools is implemented in app level python on PyPy, why not just use them instead of rewriting modules?

alanjds commented 6 years ago

Comment by ns-cweber Saturday Jan 07, 2017 at 01:19 GMT

It's MIT licensed, so that shouldn't be a concern. Seems like it would save a lot of work.

alanjds commented 6 years ago

Comment by trotterdylan Saturday Jan 07, 2017 at 04:09 GMT

Definitely support this but from my brief look at PyPy stdlibs it seemed like there was a fair bit of weirdness going on with some of the PyPy code, e.g. https://bitbucket.org/pypy/pypy/src/61294de6f130/pypy/module/itertools/?at=default

I have no problem begging, borrowing and stealing what we have to to get the stdlib functional.

alanjds commented 6 years ago

Comment by ns-cweber Saturday Jan 07, 2017 at 05:32 GMT

Yeah, that is strange. Definitely optimized for PyPy. That module in particular depends on PyPy's jit package.

alanjds commented 6 years ago

Comment by jacobh Monday Jan 09, 2017 at 02:56 GMT

Another option is https://github.com/pybee/ouroboros

alanjds commented 6 years ago

Comment by trotterdylan Monday Jan 09, 2017 at 04:34 GMT

Interesting I did not know about that project! I'm a little unclear about the origin of the libraries. Seems like some of it's from PyPy.

I'm concerned about the licensing. It looks like a non-standard license in addition to PSF: https://github.com/pybee/ouroboros/blob/master/LICENSE

Does anybody know if that project includes novel code or is it all from other places?

alanjds commented 6 years ago

Comment by tav Tuesday Jan 10, 2017 at 13:36 GMT

@trotterdylan It looks like most of it is from PyPy/Python, so you can just grab the standard licensed files.

alanjds commented 6 years ago

Comment by ulope Wednesday Jan 18, 2017 at 13:44 GMT

@trotterdylan You should ask @freakboy3742. He's the mastermind behind BeeWare (pybee)

alanjds commented 6 years ago

Comment by freakboy3742 Wednesday Jan 18, 2017 at 22:18 GMT

You can certainly use PyPy code as a starting point - but "Just Use PyPy" isn't a complete solution, because there will always be system native components. For example, you can't implement a socket library without being able to access sockets.

I've got similar needs to Grumpy on my projects pybee/batavia and pybee/voc, which is why I started pybee/ouroboros. Most of the code that is already there has originated from either PyPy or CPython.

I'd be delighted to collaborate on Ouroboros with anyone from the Grumpy team (and any other Python implementations, including PyPy) who shares the need of having a Python standard library that isn't derived from CPython.

alanjds commented 6 years ago

Comment by trotterdylan Wednesday Jan 18, 2017 at 22:49 GMT

Thanks for sharing, Russel. A collaboration sounds really interesting. A common pure-Python subset of the standard library that could be shared by all the different interpreters would make implementing Python runtimes a lot less of a chore. It reminds me a bit of the old STLPort project that was so useful when every C++ compiler had its own STL quirks and limitations.

Things get really interesting when you start to think about how to pare down the native integrations to get platform-specific features working. Like, if a runtime implements and exposes a POSIX interface to Python then most stuff should "just work".

Another area for investigation is how to enable optimization. ISTM that the design of many of the CPython libraries was dictated by what needed to be written in C. The re library is the obvious example here. If there was a more generic and flexible way to specify what ends up being implemented natively, then maybe it'd be easier for many runtimes to share the same standard library.

Anyway, happy to chat about this more. It'd be really cool if Grumpy could just use Ouroboros more or less as-is. Ideally the Grumpy repo would not include the stdlib or it would include Ouroboros wholesale rather than piecemeal from CPython and PyPy.

freakboy3742 commented 6 years ago

@alanjds There are also times where it's entirely possible to implement a module in Python, but a native implementation may give a significant speed boost (e.g., Decimal was a pure Python module in Python 3.4, but was turned into a C module in Python 3.5 for performance reasons).

Getting (and documenting) a clear definition of the "native" API surface is a key piece of the puzzle; as is defining a compliance test suite to confirm that implementations are compliant. And I agree that, ideally, Grumpy, VOC, or any other implementation wouldn't include a copy of Ouroboros, but rather "link" to it in some meaningful way - that's what both VOC and Batavia currently do; they require an Ouroboros checkout as part of their build process, and "compile" the Ouroboros sources to the "native" representation needed for their own operation.

If you've got time to hook up for a chat, I'd love to discuss the potential for collaboration here. Shoot me an email and we can coordinate details.

alanjds commented 6 years ago

Hi, @freakboy3742.

I got puzzled that you replied here, not on google/grumpy... then I realized the mailbomb I triggered with this multiple issue imports :/ Sorry for that.

About the collaboration: Sure. Will mail you as soon as I got to migrate all this issues/comments to grumpyhome/grumpy repo. (See: https://github.com/grumpyhome/grumpy/issues/8)