Pure C Modules and Packages which import from C modules

[nickelpro]

Right now there is no way (unless I missed something) to build and distribute a pure C module or a package which imports directly from a C module in __init__.py because py-build-cmake tries to read metadata from the designated python source folder before building the CMake targets.

Similarly, this prevents stubgen from generating stubs for the C modules.

I haven't experimented yet, but I don't see anything preventing us from doing things in the opposite order. Build and place everything in staging, generating metadata from staging, and then using that metadata for source distributions (which would still come from the source folders, obviously) and wheels.

Motivation: For many C modules there's no need to ship a __init__.py or any python source files at all. All you want to ship in the wheel is the final binary and some type stubs. If an __init__.py were necessary for some reason it would be something trivial like:

import sys
sys.modules[__name__] = __import__('CModule.CModule', fromlist=(None,))

In more complex scenarios, packages may want to expose some exports from a CModule in the root package:

"""I am a package with a CModule inside me"""
__version__ = "1.0.1"

from CModule import my_very_fast_function
from OtherPythonStuffThatMightAlsoImportFromCModules import other, things

Which again, isn't supported right now due to the import failing, because we haven't built CModule yet or placed everything in staging.

Given that both pure C modules and packages that want to import from them are common scenarios, py-build-cmake should support them

[tttapa]

Right now there is no way (unless I missed something) to build and distribute a pure C module or a package which imports directly from a C module in __init__.py because py-build-cmake tries to read metadata from the designated python source folder before building the CMake targets.

Could you give a concrete example?
It is true that py-build-cmake will attempt to read metadata from the Python package when using the pyproject.toml [project.dynamic] option, but this should not fail on missing modules. If it can't import the module, it will simply read the relevant metadata from the AST.

I regularly import C modules directly in the __init__.py without issues in some of the projects I maintain (for example).

Similarly, this prevents stubgen from generating stubs for the C modules.

I find that stubgen usually doesn't do a good job generating stubs for C modules, that's why I usually only use it for pure python modules.
The reason why stubgen is executed before building the CMake project is that otherwise, it might overwrite stub files generated by CMake. Running stubgen first is more flexible, because you can still run it in your CMake script if you need to (the stubgen option in py-build-cmake is mostly there for convenience).

stubgen might complain about missing imports if some of the modules are generated by CMake when executed early on, but it will usually produce decent stubs nonetheless.
I realize that this is not an ideal scenario, though, so I'm open to suggestions! (Perhaps an option [tool.py-build-cmake.stubgen.order] with values pre-build or post-build?)

Build and place everything in staging, generating metadata from staging, and then using that metadata for source distributions (which would still come from the source folders, obviously) and wheels.

The metadata is required before initiating the build: things like the package name and version are passed to CMake, so they need to be resolved earlier.
It would also require doing a full build when generating a source distribution, which I don't think is a good approach.

Perhaps a metadata hook that executes CMake in script mode could be an option here? Although unless it is unavoidable to implement some crucial features, I'm not sure if we want to take on this additional complexity.

Motivation: For many C modules there's no need to ship a __init__.py or any python source files at all. All you want to ship in the wheel is the final binary and some type stubs.

I agree that this should be supported. Let me see if I can add an option for this.

[tttapa]

Motivation: For many C modules there's no need to ship a __init__.py or any python source files at all. All you want to ship in the wheel is the final binary and some type stubs.

This is now supported in 8202449b5a1b39e7262ace8bd8860e65ead33f15, and I've added a test for it here: https://github.com/tttapa/py-build-cmake/tree/bare-c-module/test-packages/bare-c-module

The behavior is opt-in using the module.generated option:

https://github.com/tttapa/py-build-cmake/blob/7d98e97190b8ada8290865e8551b07a568c588cc/test-packages/bare-c-module/pyproject.toml#L30-L31

[nickelpro]

The rest of my report is incorrect, I was trying to implement a kind of degenerate behavior of the bare module doing something like this:

from CModule.CModule import __version__, __doc__

You're correct that by actually providing the metadata:

"""This is a doc string"""
__version__ = 1.0.0

# This works fine
from CModule.CModule import CFunc

The problem here being that flit falls back to running the module if it can't find the metadata in the AST, which fails on the import. Allowing bare modules fixes the entire use case. Much appreciated on the fast response, will test shortly

[nickelpro]

Dope this works for me

I think py-build-cmake is actually a perfect project by the way. I struggled with this 3 years ago and was never satisfied with how poorly setuptools solved this problem. I know you're just one part of the ecosystem but major kudos anyway.

Closing a completed, would appreciate a 0.2.x prerelease whenever you've got a moment.

[tttapa]

I've released 0.2.0a9.