Feature request: share wheels across Tox environments

[wkschwartz]

Superficially similar to #10, which is about downloading a pre-built wheel. What I'm looking to do is build only one wheel per {envpython}.† For example, suppose I run tests in the py37 environment on Python 3.7 and in the py38 environment on Python 3.8, and then build documentation in the docs environment. If docs uses, say, Python 3.7, then docs does not need to build a wheel, but could instead share with the py37 environment. In other words, I'd like a way for tox -e py37,py38,docs to run make-wheel twice instead of three times. (In my particular use case, skip_install=True for the docs environment is off the table because I have to compile Cython modules for Sphinx autodoc to consume.)

†I'll admit to some confusion about the relationship between basepython and {envpython}. I think what matters for which wheel an environment needs is {envpython}, but maybe I'm wrong.

Suggested interface

Ideally, tox-wheel would automatically detect the {envpython} version. There would be a per-environment option to override that detection in case, for example, I want to use different build settings (e.g., CFLAGS) in different environments. This per-environment setting might be called wheel_from (or wheel_shared_with or the like) and take the name of one or more other environments. The relationship would be implicitly bidirectional: If docs can use py37's wheel, then py37 can use docs's wheel. If wheel_from is set to the empty list, then the auto-detection of {envpython} would be turned off in favor of that environment always getting its own wheel (the current behavior), even if the environment is listed in other environment's wheel_from setting.

Implementation notes

I don't know how detecting {envpython} would work because I'm insufficiently familiar with Tox's internals. For determining which environments can share a wheels, crawl the settings for the environments that will run, and use a union-find data structure to keep track of which ones can get lumped together (you don't need the full fancy union-find algorithm because the number of environments tends to be quite small; my point here is merely that you're coming up with disjoint sets of environments to lump together, each of which gets its own wheel). Then look for any empty wheel_from settings and pull them out of any of the groups to place them on their own. For each of those groups, build a wheel.

[ionelmc]

Keying on envpython wouldn't be enough since I imagine people will specialize evironments for specific build scenarios (eg: pure python, cython). I think this shouldn't be automated (eg... key off envpython+env vars) since that can also lead to undesirable situation.

I think rather the solution lies in giving power to the user to deal with this by just having configuration - eg: this env uses this wheel.

In your situation you could just use wheel_build_env - perhaps that could be aliased to wheel_from for clarity?

[wkschwartz]

Perhaps I misunderstand how wheel_build_env works. Can it be used for my py37/py38/docs example? How so?

[ionelmc]

Try this:

[testenv]
wheel = true

[testenv:docs]
wheel_build_env = py38

[wkschwartz]

My actual configuration is more complicated than that, but this seemed to achieve my goal for tox -e py37,py38,docs.

[testenv]
wheel = true
wheel_pep517 = true
wheel_build_env =
    docs: py37
    !docs: {envname}

Where it really doesn't work is with factors: tox -e py37-stuff,docs builds once for py37-stuff and once for py37 on behalf of docs. (However, tox -e py37,docs-stuff only built once.)

Incidentally, this only works because Python 3.7 is the default Python on my system. I'm not sure how to use basepython (presumably) to select Python 3.7 explicitly for docs. Anyway, I don't really want to do that because I want docs to work if you have any Python 3 installed. This gets back to my preference that tox-wheel offer an automatic mode (at least an opt-in one via a setting). My ideal situation would be that docs works with whatever Python 3 is available and that the wheel be built for the one selected.

In the meantime, wheel_build_env may be a partial solution, but the documentation could be clearer. Currently it says

If you can produce universal wheels you might want to configure the build env so that the wheel is only built once for all the envs

This made me think you can only use wheel_build_env to build for all environments at once.

[ionelmc]

Alright, does the referenced commit adequately address your usecase?

[wkschwartz]

The changed rendered incorrectly.

Maybe you want something more like this, with the .. code-block:: ini added.

Note that you can also use ``wheel_build_env`` for situation where you have many environments for the same interpreter:

.. code-block:: ini

    [testenv:py38]
    ; regular testing

    [testenv:py38-extras]
    ; tests with optional dependencies
    wheel_build_env = py38

    [testenv:docs]
    ; docs building
    wheel_build_env = py38

Anyway, that's a good addition for now because it would have gotten me started.

I still think being able to form groups of environments or generative environment dependencies would be useful as described in the OP. I understand your hesitancy to detect things automatically, but that can be a separate question from groups/generative deps. In particular, your new docs, while a good start, don't address the missing features I mentioned here https://github.com/ionelmc/tox-wheel/issues/12#issuecomment-718229057

[ionelmc]

I just don't want to reimplement parts of pip when the problem can simply be solved with some configuration. It's not like people need to shuffle around envs all day long to require such a complex feature.