HPy: a better API for Python

Website: hpyproject.org \ Community: HPy Discord server \ Mailing list: hpy-dev@python.org

Summary

HPy is a better API for extending Python in C. The old C API is specific to the current implementation of CPython. It exposes a lot of internal details which makes it hard to:

• implement it for other Python implementations (e.g. PyPy, GraalPy, Jython, IronPython, etc.).
• experiment with new things inside CPython itself: e.g. using a GC instead of refcounting, or to remove the GIL
• guarantee binary stability

HPy is a specification of a new API and ABI for extending Python that is Python implementation agnostic and designed to hide and abstract internal details such that it:

• can stay binary compatible even if the underlying Python internals change significantly
• does not hinder internal progress of CPython and other Pythons

Please read the documentation for more details on HPy motivation, goals, and features, for example:

• debug mode for better developer experience
• support for incremental porting from CPython API to HPy
• CPython ABI for raw performance on CPython
• and others

Do you want to see how HPy API looks in code? Check out our quickstart example.

You may also be interested in HPy's API reference.

This repository contains the API and ABI specification and implementation for the CPython interpreter. Other interpreters that support HPy natively: GraalPy and PyPy, provide their own builtin HPy implementations.

Why should I care about this stuff?

• the existing C API is becoming a problem for CPython and for the evolution of the language itself: this project makes it possible to make experiments which might be "officially" adopted in the future

• for PyPy, it will give obvious speed benefits: for example, data scientists will be able to get the benefit of fast C libraries and fast Python code at the same time, something which is hard to achieve now

• the current implementation is too tied to CPython and proved to be a problem for almost all the other alternative implementations. Having an API which is designed to work well on two different implementations will make the job much easier for future ones: going from 2 to N is much easier than going from 1 to 2

• arguably, it will be easier to learn and understand than the massive CPython C API

See also Python Performance: Past, Present, Future by Victor Stinner.

What does HPy mean?

The "H" in HPy stands for "handle": one of the key idea of the new API is to use fully opaque handles to represent and pass around Python objects.

Donate to HPy

Become a financial contributor and help us sustain the HPy community: Contribute to HPy.