mypyc / mypyc

Compile type annotated Python to fast C extensions
1.75k stars 46 forks source link

Development focus areas for 2024 #785

Open JukkaL opened 3 years ago

JukkaL commented 3 years ago

I'd like us to make progress in some of these areas in 2024. Feel free to propose more areas!

Python feature support

Add support for missing Python features. At least these are important:

Robustness and safety

Examples include safe # type: ignore comments and detecting stack overflows.

Consider adapting parts of the CPython test suite. Compile more open source code as regression tests.

Peak performance

Support low-level features such as unboxed floats and packed arrays to improve the maximum performance achievable with mypyc.

Reduce overhead of other fundamental operations, such as int and list operations.

Support free threading (no-GIL CPython builds)

Now that CPython is adding free threading support, it would great for mypyc to also support it.

Multiple subinterpreters

Support multiple subinterpreters with separate GILs on Python 3.12 (#993).

Usability

Improve usability. This includes improvements to the command line interface, and error handling and messages.

Performance bottlenecks

Address some of the worst performance bottlenecks in generated code. Focus particularly on cases where performance is worse than CPython, and cases where we generate bad code for common use cases.

Compilation speed

Improve compilation speed through incremental compilation, parallel compilation, making mypyc faster, generating smaller C code, etc.

JukkaL commented 3 years ago

Focus area update

Currently loop/list/integer heavy code can be quite slow relative to lower-level languages. We are faster than CPython, but major performance improvements seem possible here.

Promising improvements include fixed-width integers, packed arrays, and the removal of redundant reference count ops and runtime type checks.

This could also help with compilation speed, since mypyc itself contains a lot of code that currently isn't very efficient (e.g. data flow analysis). If we can speed up compilation, there's more room to implement additional optimizations.

JukkaL commented 1 year ago

Updated the focus areas for 2023 (edited the original summary in place).

JukkaL commented 1 year ago

Added multiple subinterpreter support (#993) as a focus area.

JukkaL commented 5 months ago

Updated for 2024 goals. The main new goal is no-gil support (#1038) -- this seems higher priority than multiple subinterpreters, though both are promising.