Choose a Python module/package hierarchy

[bamarsha]

43 updates the PyQIR packages to use a shared namespace package pyqir (pyqir.generator, pyqir.jit, etc.). It also adds additional hierarchy to the generator modules, so that e.g. SimpleModule must be imported from pyqir.generator.module, instead of simply from pyqir.generator.

We should take another look at these two things to make sure we're following Python conventions, and are choosing the right level of organization:

1. Do we want to use namespace packages (pyqir.generator) or independent modules (pyqir_generator or pyqirgenerator)?
2. Do we want to expose a nested module structure within each package (from pyqir.generator.module import SimpleModule), a flat one (from pyqir.generator import SimpleModule), or both?

Tasks

• [x] #58
• [x] Plan for namespace packages

[bamarsha]

Particularly for question 1, we should consider the added complexity of namespace packages and the possibility that Maturin may never fully support them (see PyO3/maturin#811).

[cgranade]

For question 1, my vote is that namespace packages give a much more Python-ic feel to the API, especially once we release a wheel that depends on all three other wheels for ease of installation. If a user runs pip install pyqir or conda install pyqir, then import pyqir nicely matches that UX.

It's worth noting that as well that namespace packages have been adopted by projects as large as the Azure SDK for Python for similar reasons.

We can at least partially separate that question from https://github.com/PyO3/maturin/issues/811, perhaps, if we're OK using workarounds like having pyqir/generator/__init__.py wrap a Maturin-built native module with a name like _pyqir_internal_generator.

For question 2, I think it tends to result in a better UX to have things closer to flat (this is even an explicit part of import this), such that from pyqir.generator import SimpleModule requires users to know less about the internal structure of pyqir.generator to make use of that package. There's always a trade-off, of course, such that I have often seen submodules and subpackages used as APIs grow or if a particular submodule / subpackage has a disproportionate impact on import performance. Given the size of the current API and that there's little runtime overhead to importing the whole of pyqir.generator, I feel like the balance tends to land more at flat than nested.

[bamarsha]

For question 1, my vote is that namespace packages give a much more Python-ic feel to the API, especially once we release a wheel that depends on all three other wheels for ease of installation. If a user runs pip install pyqir or conda install pyqir, then import pyqir nicely matches that UX.

I think that to an extent this is independent from namespace packages. For example, a meta pyqir package could define its own pyqir module with an __init__.py like this:

import pyqir_generator as generator
import pyqir_jit as jit
import pyqir_parser as parser

However it would mean that there's now two ways to import things if you have the metapackage installed.

[guenp]

Do we want to use namespace packages (pyqir.generator) or independent modules (pyqir_generator or pyqirgenerator)?

I think either works, there's no specific convention that is strongly recommended as far as I know. Namespace packages are a bit more common. pyqir.generator, pyqir.jit etc. works for me.

Do we want to expose a nested module structure within each package (from pyqir.generator.module import SimpleModule), a flat one (from pyqir.generator import SimpleModule), or both?

I also don't think there are strict rules for this. If we think SimpleModule is used frequently, then I would support both from pyqir.generator.module import SimpleModule and a more shorthand version from pyqir.generator import SimpleModule. If SimpleModule is more obscure then I think just the former is fine.

[idavis]

Released in v0.3.0a1