This is WIP port of Python2 stdlib compiler package to Python3.
Motivation: to have an easily hackable Python compiler for experimenting (e.g. various optimizations, instrumentation, semantic variants, etc.)
The porting project concentrates on the conversion of AST (as provided by the builtin "ast" module) to bytecode and code objects. The original Python2 package included another important part: conversion of concrete parse tree into Abstract Syntax Tree (AST). While it would be interesting to ultimately have complete pure-Python closed loop for Python compilation, to keep this specific project maintainable, lexing, parsing, AST building are outside of its scope. Other projects are welcome to provide integrated maintainable solutions for those areas (indeed, generic/non-integrated solutions for them definitely exist).
Short-term goals:
History of Python2 "compiler" package:
Currently, the package is intended to work with CPython3.5 only. If you don't have this version handy, you can easily build a binary with a script supplied in this repository, see the next section.
python3.5 -m compiler --help
python3.5 -m compiler <input.py>
By default, the command above compiles source to in-memory code objects
and executes it. If -c
switch is passed, instead of execution, it will
be saved to .pyc
file. If --dis
is passed, code will be disassembed
before executing/saving.
The projects includes a builtin test corpus of various syntactic constructs to verify codegeneration against reference output produced by CPython3.5. Currently, the project does not include peephole optimizer, run as a postprocessing pass in CPython. This means that testing should happen against modified CPython3.5 build with the peephole optimizer disabled.
The patch is available at https://github.com/pfalcon/cpython/tree/3.5-noopt
This repository includes build-cpython-compiler.sh
helper script to
download and build it. It will produce a python3.5-nopeephole
symlink
in the top-level directory, where scripts below expect to find it.
To produce reference code generation output from python3.5-nopeephole, run:
./test_testcorpus_prepare.py
This needs to be done once. Afterwards, you can run
./test_testcorpus_run.py
to compare the output produced by this compiler package against the reference.
Beyond the simple test corpus, the tests can be a applied to the compiler code itself:
./test_dogfeed.sh
Finally, the test can be run against the entire CPython3.5 standard library:
./test_stdlib_3.5_prepare.py
./test_stdlib_3.5_run.py
(All files under Lib/ are tested. Those which don't represent valid Python3
source files are explicitly excluded (there are such, to test CPython
interpreter on them). Beyond that, there may be files excluded for which
this compiler package doesn't produce the bytecode, matching the reference
interpreter output. See the test_stdlib_3.5_run.py
file for the up to date
list.)
The source code is based on the "compiler" package from Python2 standard library. It is licensed under Python Software Foundation License v2. See complete licensing terms and details in file LICENSE.
The "compiler" package is a result of dedicated work of a number of individuals, listed below (based on the git history of the official CPython repository).
Porting of the code to Python3 and further maintenance is handled by Paul Sokolovsky.
Contributors to Python2 version of the package:
$ git clone https://github.com/python/cpython
$ cd cpython
$ git checkout 2.7
$ git log --follow Lib/compiler | grep ^Author | sed -e 's/<.*>//' | sort | uniq -c | sort -n -r
# Email addresses not included to minimize spam
143 Author: Jeremy Hylton
18 Author: Neal Norwitz
18 Author: Guido van Rossum
17 Author: Georg Brandl
11 Author: Tim Peters
9 Author: Thomas Wouters
7 Author: Neil Schemenauer
6 Author: Michael W. Hudson
6 Author: Martin v. Löwis
4 Author: Brett Cannon
3 Author: Nick Coghlan
3 Author: Antoine Pitrou
2 Author: Raymond Hettinger
2 Author: Ezio Melotti
2 Author: Christian Heimes
2 Author: Benjamin Peterson
2 Author: Anthony Baxter
2 Author: Andrew M. Kuchling
2 Author: Alexandre Vassalotti
1 Author: Serhiy Storchaka
1 Author: Phillip J. Eby
1 Author: Jeffrey Yasskin
1 Author: Gustavo Niemeyer
1 Author: Greg Stein
1 Author: Facundo Batista
1 Author: Eric Smith
1 Author: Collin Winter
1 Author: Barry Warsaw
1 Author: Amaury Forgeot d'Arc
Aka "why I took Python2's compiler package and spent all this effort on it instead of using something else".