Open patrislav1 opened 1 year ago
Hi, thanks for the good question!
I think the solution would be to customize (subclass) Pybind11Extension
. After the module is built, we can set env variables to make just built .so
/.dll
discoverable by the interpreter and generate the stubs. Unfortunately, I don't have a working example at hand.
If you found a solution, please share.
Thank you for the suggestion. In the meantime I resorted to calling pybind11-stubgen
manually, when the API of my module changes (which isn't too often right now), and checking in the stubs together with the other code. When that ever becomes too annoying I'll look into the subclassing solution.
In one of my projects, I use the following approach. First, I install the package as is, without stubs. Then I use pybind11-stubgen
to place the stubs in the right location and assemble the distribution packages with stubs for PyPI.
Note that you have to explicitly list stubs in setup.py
:
...
def find_stubs(path: Path):
return [str(pyi.relative_to(path)) for pyi in path.rglob("*.pyi")]
package_name = ... # name of your package
setup(
name=package_name,
...,
# Add `py.typed` and `*.pyi` files
package_data={package_name: ["py.typed", *find_stubs(path=Path(package_name))]},
)
Example of package-distributions.sh
script:
#!/bin/bash
# Show commands
set -x
# Exit on the first error
set -e
MODULE_NAME="..." # e.g. `my_package._core`
# Could be either name of your package (same as in `setup.py`)
# or its binary part only (e.g. `my_package._core`)
# if you don't want to duplicate pure python parts of your library in `*.pyi` files
# Replace '.' with '/'
MODULE_PATH=$(echo "${MODULE_NAME}" | sed 's/\./\//g' -)
# Install the package from sources
pip install .
# Make sure required tools are installed
pip install black isort pybind11-stubgen
# Generate stubs and apply formatting
pybind11-stubgen "${MODULE_NAME}" -o ./ --numpy-array-wrap-with-annotated --exit-code
black "${MODULE_PATH}"
isort --profile=black "${MODULE_PATH}"
# Assemble final packages
python setup.py sdist bdist_wheel
Currently, the limitation of pybind11-stubgen is that the package must be importable using import
to generate the stubs. We can utilize PYTHONPATH Mechanism to achieve "executing import without following the package structure." Below is a part of my project configuration:
class PostInstallCommand:
def run(self, install_lib):
self.py_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'mpool'))
self.build_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'build'))
self.target_dir = os.path.join(install_lib, 'mpool')
self._install_lib = install_lib
self.copy_python_file()
self.copy_shared_library()
self.generate_stub()
def generate_stub(self):
env = os.environ.copy()
if 'PYTHONPATH' not in env:
env['PYTHONPATH'] = os.path.abspath(self._install_lib)
else:
env['PYTHONPATH'] += ':' + os.path.abspath(self._install_lib)
subprocess.check_call(['pybind11-stubgen', 'mpool', '-o', self._install_lib, '--ignore-all-errors'], env=env)
I'm using the setuptools method to build/install my pybind11-based module. I'd like to integrate pybind11-stubgen into the
setup.py
, so it creates and installs the.pyi
after the pybind11-based module is built. I tried to set up a post-install function using thesetuptools.command.install
override (cmdclass={'install': new_install}
). But at this point, the wheel is built but the module is not installed yet - so when I run pybind11-stubgen, it will not find the module to annotate.I don't know the Python packaging internals well enough, but this doesn't look like a very exotic problem to me, so maybe someone has already figured it out?