[BUG] FileNotFoundError jaraco/text/Lorem ipsum.txt when importing pkg_resources

[shk3]

setuptools version

71.0.0

Python version

Python 3.9

OS

Ubuntu Linux & macOS

Additional environment information

No response

Description

This seems to be a regression of #3072. My suspicion is commit d03cd0e6e21067e2802d811d5a4535f69da80788 might play a part of it, but I don't know enough about Python's package management to be sure.

We are building with an older version of Sphinx with Bazel, which effectively import pkg_resources just like what has happened in #3072, which is supposed to have been fixed via #3077.

Expected behavior

Pinning setuptools==70.3.0 in requirements.txt works fine. This issue started with setuptools==71.0.0 as well as setuptools==71.0.1.

How to Reproduce

Pretty much the same as #3072, except in our case we were building an old Sphinx 4.1.0, which doesn't import pkg_resources anymore in new versions. We are reporting because this could happen to other projects that import pkg_resources in a later version.

Using Bazel and rules_python, build a zip executable using:

bazel build //<target> --output_groups=python_zip_file
python3 bazel-bin/<target>/<target>.zip 

Output

Traceback (most recent call last):
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/repo/<file>.py", line 8, in <module>
    from sphinx.ext.apidoc import main as sphinx_api_doc
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/ext/apidoc.py", line 30, in <module>
    from sphinx.cmd.quickstart import EXTENSIONS
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/cmd/quickstart.py", line 40, in <module>
    from sphinx.util.template import SphinxRenderer
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/util/template.py", line 22, in <module>
    from sphinx.jinja2glue import SphinxFileSystemLoader
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/jinja2glue.py", line 20, in <module>
    from sphinx.application import TemplateBridge
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/application.py", line 43, in <module>
    from sphinx.registry import SphinxComponentRegistry
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/registry.py", line 24, in <module>
    from pkg_resources import iter_entry_points
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__setuptools/site-packages/pkg_resources/__init__.py", line 93, in <module>
    from jaraco.text import (
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__setuptools/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 231, in <module>
    files(__name__).joinpath('Lorem ipsum.txt').read_text(encoding='utf-8')
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/external/python3_9_x86_64-apple-darwin/lib/python3.9/pathlib.py", line 1266, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/external/python3_9_x86_64-apple-darwin/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/external/python3_9_x86_64-apple-darwin/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__setuptools/site-packages/setuptools/_vendor/jaraco/text/Lorem ipsum.txt'

[osfrickler]

I think this should be fixed by the just released 71.0.2, can you check against that?

[allenbenz-doma]

Seeing the same issue, wasn't fixed for me in 71.0.3.

Writing down more context to save someone else time.

bazel doesn't support whitespaces in paths There's an experimental flag that reportedly can be used if you're using bazel 7.0.0 or above.

The previous workaround looks like it was removed in this PR Specifically this commit and this other commit removed the a regex filter that was deleting the bit of code loading the Lorem ipsum.txt file in the vendored copy of jaraco.

jaraco is unlikely to rename the file.

[jaraco]

Thanks for the report and detailed analysis. And sorry for the inconvenience while Setuptools works to clean up some cruft that's accumulated.

I'd forgotten that I'd built in a patch for jaraco.text when vendorizing it. I was really happy to be rid of all that tech debt in 9234fc35f37bd74a32501f22c698f6a8af26ffed, but I'd been working from the assumption that the patches were there solely to support vendorizing and not there to support other issues, so the removal of the workaround introduced in #3077 was unintentional.

As setuptools moves toward a regime that doesn't depend on vendored packages, any workarounds that apply to the vendored package will become less viable. Setuptools 71 has a preference for naturally-installed packages. What happens in a Bazel-managed environment if jaraco.text is installed naturally?

bazel doesn't support whitespaces in paths There's an experimental flag that reportedly can be used if you're using bazel 7.0.0 or above.

That's great news! I'd really like to know (a) if employing that flag fully addresses the issue, and (b) how viable it would be for users to adopt Bazel 7.

Assuming that's not a path to a long-term solution, is there another path by which setuptools and/or jaraco.text can be patched on import to the Bazel environment? I'd like for Setuptools not to have to bear the responsibility for compatibility with a downstream tool. I know at Google, which uses a similar tool, the jaraco.text was imported with a patch that renames the file to make it viable in that environment. Is there a way to do something similar for //target?

[shk3]

I can confirm that this issue still exists in 71.0.2 and 71.0.3. We are running Bazel 6.4.0. I'm not sure about what the effort would be for upgrading, so I'm leaving it for other folks who might be impacted by this to tell.

As for my particular use case, we were able to fix by upgrading the Sphinx version, which does not import pkg_resources anymore. So, I'm no longer impacted.

[w4-donguk]

I can confirm that this issue still exists in 71.0.2 and 71.0.3. We are running Bazel 6.4.0. I'm not sure about what the effort would be for upgrading, so I'm leaving it for other folks who might be impacted by this to tell.

As for my particular use case, we were able to fix by upgrading the Sphinx version, which does not import pkg_resources anymore. So, I'm no longer impacted.

which version did you updated to?

[shk3]

@w4-donguk We upgraded Sphinx from 4.1.0 to 7.4.5. They removed the import of pkg_resources since 4.4.0.

FWIW, the Sphinx upgrade only resolved our issue by not importing pkg_resources. For other modules that really need to import pkg_resources, they would still encounter this issue with the latest setuptools with Bazel.

I'm reporting mainly to make sure the community is aware of this unexpected side-effect. I agree with @jaraco that setuptools probably shouldn't really compromise for all kind of weirdness in the downstream systems. I personally haven't really got the time to dive into Bazel to see if there could be patches on Bazel rules to get it work as folks suggested.

[jaraco]

I'd prefer to do nothing here and to use the package naturally, and rely on downstream tooling to account for patching any jaraco.text or to employ the proper fix using the latest Bazel, especially because the latest Setuptools no longer has control over which jaraco.text is used (as it will prefer a site-installed copy to its own).

Actually, that gives me another idea for a workaround - if bazel (or the tooling around it) had a way to install a patched version of jaraco.text (either without that property or with it renamed), Setuptools would use that version and not its bundled copy. One could package that up as jaraco.text-bazel-safe and install it when installing Setuptools. It's ugly, but at least it places the maintenance responsibility nearer the cause.

I'm going to close this for now, but feel free to follow-up, especially if the proposed workarounds turn out to be too onerous, and we might consider re-introducing the patch of the vendored copy.