search

we-like-parsers / pegen

PEG parser generator for Python
https://we-like-parsers.github.io/pegen/
MIT License
161 stars 33 forks source link

Publishing on the PyPI #1

Closed isidentical closed 3 years ago

isidentical commented 3 years ago

Pegen's Python parser generator is really cool, and also very useful though the version on the PyPI seems not sync with the upstream version under Tools/peg_generator. It is possible now to just clone CPython and install it though not very feasible. Or the other way would be, after generating the parser just hosting pegen/parser.py / pegen/tokenizer.py under the project (which is what I do now) though it is not a very good solution either. I think it would be really amazing if we host this package on the PyPI (maybe with automating some stuff on the GH actions to clone CPython, check if anything changed under Tools/peg_generator, and sync it with here and open an issue to notify the maintainers that they should make a release). If any help needed, I can try to assist.

pablogsal commented 3 years ago

This is indeed some of the action items from the core Dev sprint. With all that has been happening it has been impossible to find time for it but I will try to do it soon.

One limitation is that the C generator is not that useful outside C Python because the AST symbols are not exposed, so we can only expose the Python generator.

lysnikolaou commented 3 years ago

I agree that exposing only the Python generator is the way to go. It's both less work maintaining it for us and the value of exposing the C generator is really really minimal.

lysnikolaou commented 3 years ago

Should we maybe start a new repository and move the version we want to publish there? With all the blog post code and the C generator code here, it feels that a new repo would be much cleaner.

isidentical commented 3 years ago

I think it makes sense (to preserve the blog posts / sources)

pablogsal commented 3 years ago

Should we maybe start a new repository and move the version we want to publish there? With all the blog post code and the C generator code here, it feels that a new repo would be much cleaner.

Appart from the blog post, is there any reason we want to preserve the rest of the code in this repo? The canonical version of all the code itself lives in CPython now and much if the stuff we have here currently is out of date.

MatthieuDartiailh commented 3 years ago

I would also be interested in the Python generator since I maintain a project that defines a superset of Python (https://github.com/nucleic/enaml). Currently it relies on ply which is LALR1 parser and may not be able to keep up with the new language features making use of the full power of the parser.

pablogsal commented 3 years ago

I renamed the old repo to https://github.com/we-like-parsers/pegen_experiments and I have created https://github.com/we-like-parsers/pegen with some initial scaffolding, also updating the code from CPython

lysnikolaou commented 3 years ago

Should I start removing stuff? The way I see it, we probably can remove anything not related to the python generator. docs, some of the scripts and some of the tests can certainly go. We can wrap the thing around a nicer API/command-line utility next and maybe improve the README?

gvanrossum commented 3 years ago

Do I need to fix the links in my blog posts on Medium? Some of those link to the examples in this repo.

pablogsal commented 3 years ago

Do I need to fix the links in my blog posts on Medium? Some of those link to the examples in this repo.

Oh, good point! I will move the files back so the links are still valid.

pablogsal commented 3 years ago

Should I start removing stuff? The way I see it, we probably can remove anything not related to the python generator. docs, some of the scripts and some of the tests can certainly go. We can wrap the thing around a nicer API/command-line utility next and maybe improve the README?

I did some general removal already and I prepared the package with some stuff for easier testing of the pypi packages, but feel free to continue improving and cleaning stuff :)

pablogsal commented 3 years ago

Do I need to fix the links in my blog posts on Medium? Some of those link to the examples in this repo.

Interesting. I checked the links in your blog and GitHub seems smart enough to redirect to the previous (now renamed) repo!

lysnikolaou commented 3 years ago

This has been there since 2013! https://github.blog/2013-05-16-repository-redirects-are-here/

MatthieuDartiailh commented 3 years ago

The first release is online since September 7th