bytecode compilation flag doesn't do anything

[peekxc]

As far as I can tell, meson-python doesn't support bytecode compilation of python modules inside of wheels.

In a simple test package, I tried adding the bytecompile flag via pyproject.toml:

# pyproject.toml
[build-system]
build-backend = 'mesonpy'
requires = ['meson-python', 'wheel', 'ninja', 'numpy']

...

[tool.meson-python.args]
setup = ['-Dpython.bytecompile=2']

Then I tried building the wheel with build via python -m build -nx -v, which does display the option as set during the build step:

skpackage 0.0.2

  User defined options
    Native files      : /private/var/folders/0l/b3dbb2_d2bb4y3wbbfk0wt_80000gn/T/build-via-sdist-a0tojhrv/skpackage-0.0.2/.mesonpy-9u1xqvsd/meson-python-native-file.ini
    buildtype         : release
    default_library   : static
    python.bytecompile: 2
    b_ndebug          : if-release
    b_vscrt           : md

However, when I unpack the corresponding wheel, there's no pycache directory or *.pyc files in sight. Incidentally, when I install the wheel into e.g. a virtualenv, I notice that bytecode files are there:

├── __init__.py
├── __pycache__
│   ├── __init__.cpython-311.pyc
│   └── __version__.cpython-311.pyc
└── __version__.py

Does this behavior sound normal? I would've thought specifying the byte-compilation flag would make meson add them directly to the wheel, but this doesn't seem to be the case

(I'm using meson 1.5.2 btw)

[eli-schwartz]

I believe meson-python intentionally skips this, most likely because byte compiled caches are quite version specific so having them inside of a wheel that could be installed on any python version is potentially problematic.

Meson has the flag because it can be used to install the modules directly to the system, in which case it takes the role of pip and runs byte-compilation immediately for you.

[peekxc]

I see. So in terms of its functionality, is the flag then not intended for e.g. version-specific wheel distribution on PyPI?

I'm generating wheels with extension modules + python files which are Python version specific (e.g. pkgname-cp311-macosx_10_16_x86_64.whl). At least, these are specific to the OS / ISA and the minor version of Python, not including the patch or finer subversions. Since I'm already targeting minor versions per wheel, I figured I would add bytecode compilation; but you're saying this should really happen on the actual computer the package is installed to?

[eli-schwartz]

I personally have no objection to the idea you suggest here. Speaking as a linux distro developer, we do ship ahead of time compiled bytecode, just like you want to do. But we don't use wheels at all.

I think standards groups for python don't really distinguish this case (having C extensions is a bit of a blind spot for the standards, in my experience) and they would probably assume you should not do it, which feels slightly awkward in my view.

[dnicolodi]

requires = ['meson-python', 'wheel', 'ninja', 'numpy']

Why do you list wheel and ninja as build dependencies? wheel should not be needed and ninja is requested by meson-python if it is not provided on the system.

Does this behavior sound normal?

Yes, things are working as expected.

I would've thought specifying the byte-compilation flag would make meson add them directly to the wheel, but this doesn't seem to be the case

Why would you like to included the bytecompiled Python source in the wheel? The wheel installer is going to byte compile the Python source code again anyway, thus having the bytecode in the wheel is not doign anything other than wasting some storage space at best, and is causing the installer to fail at works because of a collision in the bytecode filename with an installed file.

[peekxc]

Why would you like to included the bytecompiled Python source in the wheel?

I was originally looking for ways to improve module import time for a package, when I stumbled upon this; I was just thinking of bytecode compilation as analogous to compiling extension modules, and thought I might try to save the parser some time. It just seemed like a natural thing to add to the CI pipeline since I was already compiling code for platform-and-version-specific wheels.

The wheel installer is going to byte compile the Python source code again anyway

Ah I didn't realize this actually, I thought it was done at import time.

---

Anyways, it seems my issue is resolved

[eli-schwartz]

The wheel installer is going to byte compile the Python source code again anyway

Ah I didn't realize this actually, I thought it was done at import time.

It is done at import time if it hasn't been done previously ;) and it's possible to disable the default wheel installer behavior of performing byte compilation. (Just as it's possible to disable the meson behavior of performing byte compilation.)

Particularly for Linux distros, doing it at import time would be a major problem since there is a package manager! And the package manager has to track all files, so having files in root owned directories created later on by running applications would cause big issues. Also, if the program isn't run as root it wouldn't be able to perform automatic on-the-fly byte compilation anyway. (Technically, it does byte compilation as part of importing, but cannot save the results to the cache, so byte compilation has to be done first thing in every application startup.) So we have to do it ourselves, which is also fitting since we aren't a wheel installer -- we are a general purpose package manager.

The fact that it improves the speed of the first-time run of the package is a convenient bonus within that use case.