search

hpyproject / hpy

HPy: a better API for Python
https://hpyproject.org
MIT License
1.1k stars 52 forks source link

no hpy/__init__.py? #63

Open arigo opened 4 years ago

arigo commented 4 years ago

Is there a reason for why there is no hpy/__init__.py file, turning hpy into a "namespace"? Apparently that triggers a bug in "setup.py install" that seems not to handle this properly. I'd vote for avoiding namespaces if setuptools doesn't reliably support them.

antocuni commented 4 years ago

The original PR/discussion is this: https://github.com/pyhandle/hpy/pull/16

TL;DR: we need it to be a namespace if we want to distribute hpy.universal and hpy.devel (and possibly more) separately. I think there are already other projects using this structure.

What is the problem you are experiencing?

arigo commented 4 years ago

I didn't check but jtiai on irc said "looks like there is some kind of a bug in setup.py install".

antocuni commented 4 years ago

I'm not opposed to refactor/change things, but I'd like something more than that as a reason 😅

arigo commented 4 years ago

According to https://setuptools.readthedocs.io/en/latest/setuptools.html#namespace-packages, we're not doing the right thing. This docs says we must put an __init__.py, but one with a specific line in it (I didn't study this in detail).

antocuni commented 4 years ago

I think that that link refers to "pkg_resources-style", the "old" way of doing namespaces, which is compatible with Python2. This link summarizes the possibilities. Indeed, it seems to imply that "native namespace packages" do no work with setup.py install, although I don't know the details. For reference, see also PEP 420 for why they think native namespaces are a good idea.

So, I think we have two choices:

  1. keep using native namespaces, and possibly emit a warning/error which tells you to use pip in case you use setup.py install (hint: you can always do pip install .)

  2. switch to pkgutil-style namespaces.

That said, I am fine with either solution, but I suggest we should ask to someone who knows more about this to get all the implications.

mattip commented 4 years ago

@ncoghlan was one of the authors, maybe we could get a response or a referral to someone else who could help out.

uranusjr commented 3 years ago

Since setup.py install is strongly disfavoured, it is probably better to discourage it as much as possible.

This also bring out a related issue; HPy currently relies quite heavily on setuptools and distutils, which is an approach the packaging community is actively moving away from. This is not an issue for proof-of-concept works (where setup.py build_ext is really the only packaging command that needs to work), but would cause an issue when a more general audience look to try HPy out, where they’d want to install HPy via a wheel, and create PEP 517 wheels from extensions built with HPy, etc.

I’m currently pretty occupied myself, but intend to try some modern packaging strategies on HPy itself, and the extensions it builds. HPy is an awesome project, and this is the best I can do to help.

mattip commented 3 years ago

[I] intend to try some modern packaging strategies on HPy itself ...

That would be awesome. Are there models of modern packaging strategies for building c-extensions you would recommend looking at for inspiration?

uranusjr commented 3 years ago

Setuptools is actually a pretty good (the best?) choice for building the extension modules, since it contains a good amount of practically-working compiler and dependency introspection logic. Alternative build systems are unforauntately not very actively pursued afaik since setuptools.Extension is “good enough” for most people, which is not unreasonable.

The approach that may be beneficial is to create an interface to stand into setuptools.Extension, so other packaging tools can use the same interface to build both old- and new-style extension libraries. Pdm and Enscons have made some efforts interfacing with setuptools.Extension, but more work is needed.

mattip commented 3 years ago

Maybe we should take this offline, but just summarizing the rabbit hole your links sent me down:

There are probably more

It is not clear to me how HPy would create an interface that would allow it to be used easily with all these tools. Here are a few examples of ways other projects interfacing compiled code have dealt with this:

antocuni commented 3 years ago

@uranusjr thank you for your feedback, your help will surely be welcome.

However, I am not sure to understand what is the exact problem you are trying to solve (which doesn't mean that the problem does not exist of course). In test_pof.sh we test three different installation strategies:

  1. setup.py bdist_wheel + pip install https://github.com/hpyproject/hpy/blob/cc98cac7f623572c4b0096c1767d738b57b4beae/proof-of-concept/test_pof.sh#L72-L89
  2. setup.py install
  3. setup.py build_ext -i

If I understand it correctly (and I'm not sure), (1) is all we need for "modern" installation tools. The fact that we also support (2) and (3) is a convenience but should not harm anyone.

Another thing to keep in mind is that we want to make the transition from Python/C to HPy as easy as possible, and the vast majority of current extensions are written with setup.py. That's why we introduced hpy_ext_modules

uranusjr commented 3 years ago

I should have provided more context from the beginning 🙂 I wrote the initial comment mainly to answer the namespace module question (no, don’t bother with setup.py install), and did not expect to get into this much detail this quickly.

A common subtext around most recent development in Python packaging is to avoid the coupling (both build-time and run-time) to the distutils/setuptools stack. This is actually kind of analogous to what HPy aims to improve—there are certain things the default stack (CPython for HPy, distutils/setuptools for packaging) does not do well, and we want to provide alternatives to solve those issues. But the interface the default stack provides is not adequate for alternatives to implement, so we want to introduce better abstractions for people to more easily interface with multiple solutions.

HPy’s current build system has a hard dependency on distutils and setuptools, and interfaces with them in a way alternative packaging tools cannot reuse. This means those solutions cannot support building HPy extensions without rewriting most of the tool stack. So what I’m trying to solve is to look into possible improvements in this area early on, to avoid HPy becoming another thing that prevents people from using alternative packaging solutions.

antocuni commented 3 years ago

HPy’s current build system has a hard dependency on distutils and setuptools, and interfaces with them in a way alternative packaging tools cannot reuse. This means those solutions cannot support building HPy extensions without rewriting most of the tool stack. So what I’m trying to solve is to look into possible improvements in this area early on, to avoid HPy becoming another thing that prevents people from using alternative packaging solutions.

sounds good to me. Note that it's already partly like that, because most of the logic is inside hpy.devel.HPyDevel, which is used by both the testing infrastructure and the setuptools integration. However, HPyDevel is based on setuptools, so you probably want to decouple the logic there.

Feel free to come to the IRC channel and/or write to the mailing list and/or open an issue to discuss more concrete details.

cclauss commented 10 months ago

https://github.com/hpyproject/hpy/issues/63#issuecomment-812452265

The fact that we also support (2) and (3) is a convenience but should not harm anyone.

It seems to break the GitHub Actions (see #467) and harms users on macOS v12, v13, v14.