hpyproject / hpy

HPy: a better API for Python
https://hpyproject.org
MIT License
1.1k stars 52 forks source link

Improve the integration with setuptools and the packaging ecosystem #370

Open antocuni opened 2 years ago

antocuni commented 2 years ago

As described below, the current HPy/setuptools integration is a mess and could/should be vastly improved. And in general, we should try to engage more with the broader setuptools/pypa/packaging community. This issue wants to be a summary of the current status AND a set of open topics/questions that we can use as a starting point to talk to them.

Summary of current behavior of HPy+setuptools

From the user point of view:

Currently, we implemented the behavior described above by using a mixture of proper features and horrible monkey-patching (sorry for that). The relevant code is in hpy/devel/__init__.py: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py

Notable implementation details:

hpy_ext_modules is implemented using an setuptools entry point: this is our entry point to do all the the rest: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L133-L147

The horrible monkey patching happens here: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L78-L117

Our moneky-patched build_ext uses finalize_options as a hook to be executed "at some point". In that hook, we take all the hpy_ext_modules, modify them in various ways and add them to the "normal" list of ext_modules: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L308-L320

The Extension instances which were in hpy_ext_modules are modified in various ways: in particular we add the relevant include_dirs, sources and macros: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L290-L306

In order to produce a binary whose extension is .hpy.so, we do another terrible hack: this is the code https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L330-L341

The horrible hack comes from the implementation of is_hpy_extension, because at this point in time hpy_ext_module=[...] and ext_modules=[...] are mixed together. The only way that we found was to wrap the extension name into a special subclass of str :facepalm:. We tried other ways (e.g. setting an is_hpy attribute on the extension itself) but they didn't work because the modifications are lost somewhere in the internals of distutils/setuptools: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L181-L204

Writing the foo.py stub for universal mode was also tricky: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L343-L386

And finally, we had to modify get_export_symbols in order to work on Windows: this is needed because the HPy universal ABI exports a function called HPyInit_*, not PyInit_*: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/__init__.py#L388-L398


Removing the hacks

We would like to work with the general community to find a set of hooks and features that will allow us to implement the desired functionality without using those horrible hacks, so we would like to input from the setuptools community and see what is their stance on it.

At the beginning I thought that we could propose a set of hooks to solve our needs, but then I realized that they surely have a better idea of what is doable, what is "good" and what is "bad", so probably it's better if they are the ones who propose the hooks/API/interface/whatever.

Note that from our point of view they end result doesn't necessarily need to be the very same as it is now. For example, currently we use hpy_ext_modules=..., but I would be very happy even with e.g. ext_modules=[Extension('foo', ..., hpy=True)] or someething like that, as long as setuptools calls us in a "clean" way.


Other build systems

I know that there are other build systems other than setuptools, but personally I don't know anything about them. I suppose we should have a story for people who wants to use HPy without setuptools: e.g. some better defined API to tell the world what are the include dirs, the extra sources, the resulting filename, etc. etc.


Filename extension

This is another open topic: currently we produce foo.hpy.so because it was the simplest thing to do, but eventually we would like to have official support by the community and the ecosystem. There are two easy thing which we can do now:

This would result in something like: foo.hpy-1-x86_64-linux-gnu.so.

Note for external readers: one cool thing about HPy is that the ABI is extensible by design, so once we finalize hpy-1 it is very likely that it will stat stable for years and years. But of course it is good to have a version number so that we can be ready if we will ever need to change something in the future.


PyPI/wheels support?

This is a direct consequence of the section above: what needs to happen to be able to have hpy wheels, uploaed them to PyPI and then pip install them?

timfel commented 1 year ago

Discourse thread: https://discuss.python.org/t/how-to-make-setuptools-work-correctly-with-hpy/19975/4