PEP 269 Implementation

[064805c8-721f-4156-b40d-ff860d76b595]

BPO	599331
Nosy	@gvanrossum, @loewis
Files	pep269.tar.gz: Gzip'ed tar file with patch diffs, extension modules (new to core), and examples/tests. pep269.tar.gz: Update to previous u/l.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core'] title = 'PEP 269 Implementation' updated_at = user = 'https://bugs.python.org/jriehl' ``` bugs.python.org fields: ```python activity = actor = 'jriehl' assignee = 'none' closed = True closed_date = None closer = None components = ['Interpreter Core'] creation = creator = 'jriehl' dependencies = [] files = ['4532', '4533'] hgrepos = [] issue_num = 599331 keywords = ['patch'] message_count = 15.0 messages = ['41006', '41007', '41008', '41009', '41010', '41011', '41012', '41013', '41014', '41015', '41016', '41017', '41018', '41019', '41020'] nosy_count = 3.0 nosy_names = ['gvanrossum', 'loewis', 'jriehl'] pr_nums = [] priority = 'low' resolution = 'rejected' stage = None status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue599331' versions = ['Python 2.3'] ```

[064805c8-721f-4156-b40d-ff860d76b595]

The following are files for the implementation of PEP-269. The primary changes to the core involve moving required pgen modules from the pgen only module list to the Python library module list in Makefile.pre.in. Some of the modules required better memory allocation and management and the corresponding deallocators are in the Python extension module (maybe these should be moved into the pgen modules in Parser).

Initially included are two implementations. The first is a basic implementation that follows the PEP API more or less. The second (currently unfinished) implementation provides a more object oriented interface to the extension module.

Please note there are some commented out modifications to setup.py as I was unable to build the extension module(s) automagically. For some reason the linker I was using (on a BSD box) complained that it couldn't find the _Py_pgen symbol, even though I verified its existence in the new Python library. Maybe it is checking against an older Python library on the system?

Things to be done (as of initial submission) include resolving on a single interface, documenting the one true interface, and finishing any unimplemented routines (such as are found in the OO implementation).

In the final integration, a pgenmodule.c file should be added to the Modules directory in the main distribution.

[gvanrossum]

Logged In: YES user_id=6380

I guess I'm going to hve to look at this to pronounce...

[gvanrossum]

Logged In: YES user_id=6380

I looked a bit, and it's very raw - the author admits that. Jon, are you still working on this? Do you have a more polished version?

Maybe we can separate out the memory allocation patches and commit these already? They don't break anything.

[064805c8-721f-4156-b40d-ff860d76b595]

Logged In: YES user_id=22448

Guido, as per my private message, I'll attempt to submit another patch by the end of the month, pending resumption of "work" on the 23rd. Commitment of the memory allocation patch is fine, and any future patches would be against the updated pgen code (I don't have commit permissions, so someone else will have to do this possibly.)

[gvanrossum]

Logged In: YES user_id=6380

Cool. Maybe I'll get to it, maybe not. :-(

[gvanrossum]

Logged In: YES user_id=6380

Lowering priority until Jon has his next version ready.

[gvanrossum]

Logged In: YES user_id=6380

Closed. Jon, if you're still interested in this, upload something new and reopen the patch (or I can do that).

[gvanrossum]

Logged In: YES user_id=6380

OK, reading the README now...

[gvanrossum]

Logged In: YES user_id=6380

I've checked in the changes to the parser and Makefile etc. This means you should be able to distribute pgenmodule.c as a clean extension to Python 2.3.

I've decided not to incorporate pgenmodule.c at this time. There are three problems with it as I see it:

• The C coding style is really different, and makes it hard to read: 2-space indents instead of tabs, spaces before left parentheses in argument lists, space between and name in things like "PyObject name", placement of curly braces, reversal of arguments to comparisons (e.g. "NULL != x"), and long strings of underscores in comments.

• The code isn't robust enough: I got a segfault (can't remember after what sequence of events, sorry), and it caused a SystemError (error return without exception set) when the grammar file couldn't be read.

• The pgen API seems a bit strange -- some things that I would expect to be methods of AST objects are functions in the pgen module (even when a companion function *is* an AST method).

• The unit test doesn't fit in the Python test framework.

• I cooked up a little expression grammar and tried to parse a sample string, but all I got was exceptions. I guess I am missing something, but there's no tutorial that explains me how to do things (I was copying the code from test_pgen.py which works :-( ).

Nevertheless, thanks for the work so far -- I'm sure at some point pgen will be usable from Python!

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Logged In: YES user_id=21627

Jon, are you still interested in taking this further?

[064805c8-721f-4156-b40d-ff860d76b595]

Logged In: YES user_id=22448

I am to some extent. I've done several wacky things, namely putting the pgen extension module into my Basil project (basically I think PEP-269 is dead), as well as re-written pgen in "pure" Python (no C extension module calls) The pure Python pgen looks like it will be going into Pypy. I don't know if I am interested in the speed enough to warrant serious analysis of the pgen C extension and the segmentation fault Guido reported. Let me know if you had anything specific in mind.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Logged In: YES user_id=21627

I was wondering what the possible next steps could be. With your current analysis, it appears that

1. declare PEP-269 withdrawn
2. reject this patch would be a possibility. I would want to keep this open only if some volunteer would indicate interest in completing this work.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Logged In: YES user_id=21627

I have now marked the PEP as deferred, until some volunteer picks it up again. I'm rejecting this patch (but left a pointer in the PEP for reference).

[gvanrossum]

Logged In: YES user_id=6380

FWIW, I *also* rewrote pgen in Python. Is the Pypy version real? If so, I ma not have to publish my code (which is owned by Elemental Security but if there's demand I can attempt to open-source it).

[064805c8-721f-4156-b40d-ff860d76b595]

Logged In: YES user_id=22448

It may not be super usable (I believe it is as usable as pgen is, which is just usable enough to automate Python parser generation), but I consider it finished. The SVN URL is now:

http://codespeak.net/svn/basil/trunk/basil/common/python/