Error trying to use pyspark

[njlr]

When I try to use the pyspark package, I get this error:

Backend 'setuptools.build_meta:legacy' is not available.

Here is the pyproject.toml:

[tool.poetry]
name = "cowapp"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "3.10.11"
cowsay = "^6.1"
numpy = "^2.0.1"
pyspark = "^3.5.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Full repro code is here: https://github.com/njlr/rules_pycross_oci/tree/issue-pyspark

Trigger the error with:

bazel build //...

[jvolkman]

Unfortunately build dependencies are not included in the poetry lock file, so you'll need to include them yourself like here. You'll also need to add dependencies on setuptools and wheel to pyproject.toml and re-lock.

[njlr]

Unfortunately build dependencies are not included in the poetry lock file, so you'll need to include them yourself like here. You'll also need to add dependencies on setuptools and wheel to pyproject.toml and re-lock.

Thanks for the pointers.

I made some progress but hit a new error:

ERROR Backend 'mesonpy' is not available.

Repro is in https://github.com/njlr/rules_pycross_oci/tree/issue-pyspark

[jvolkman]

It's the same issue: you'd need to add an explicit dependency on mesonpy.

But meson is a full-fledged build system, so I'm not sure that attempting to build a wheel with it will be very successful nested within a Bazel invocation. It's been on my TODO list for a while to add support for other build systems like meson, maturin (for rust-based wheels), etc. Currently the wheel builder in this rule set assumes setuptools and distutils. These are by and large still the standard, though, so maybe mesonpy will also work with the same assumptions. But I've never tried.