Python 3.11: venv pythonpath seems to conflict with importlib

[Yun-Kim]

Riot seems to have problems with Python 3.11's importlib in terms of managing virtual environments. Specifically in dd-trace-py using riot, Hypothesis' vendored text file is not being found even though there is no problem running ddtrace.tests.tracer.test.http.py tests locally with Python 3.11.

Steps to Reproduce

Package Structure

To reproduce, here is my file structure:

repro_script.py
repro_package
├── setup.py
└─── src
      ├─ __init__.py
      ├─ test.py
      └─ vendor
          ├─ __init__.py
          └─ domains.txt

repro_script.py:

from repro_package import test

repro_package/setup.py:

import os
import setuptools

def local_file(name):
  return os.path.relpath(os.path.join(os.path.dirname(__file__), name))

SOURCE = local_file("src")

setuptools.setup(
  name="repro_package",
  packages=setuptools.find_packages(SOURCE),
  package_dir={"": SOURCE},
  package_data={"repro_package": ["vendor/domains.txt"]}
)

repro_package/src/test.py:

from importlib.resources import files

domains_file = files("repro_package.vendor") / "domains.txt"
domains = domains_file.read_text().splitlines()

repro_packages/src/vendor/domains.txt:

APPLE
BANANA
CHERRY

Reproduction Command

python --version
# Python 3.11.0rc2+

python -m venv .venv
source .venv/bin/activate
pip install -e repro_package
python repro_script.py

[Yun-Kim]

Note: we caught this issue when running tracer (ddtrace) with riot, using Python 3.11. The workaround was to move the test_http.py tests to a separate tox command, but we can still reproduce the problem by reverting the --ignore=tests/tracer/test_http.py argument in the riot tracer venv.

[Yun-Kim]

Specifically, the symptom we're seeing from running in Python 3.11 is easily reproduced by running the following commands, with hypothesis install via pip:

from importlib.resources import files
print(files("hypothesis.vendor"))
print(files("hypothesis.vendor") / "tlds-alpha-by-domain.txt")

Without using riot, the output is:

PosixPath('/Users/yun.kim/go/src/github.com/DataDog/virtual_envs/py3_11/lib/python3.11/site-packages/hypothesis/vendor')
PosixPath('/Users/yun.kim/go/src/github.com/DataDog/virtual_envs/py3_11/lib/python3.11/site-packages/hypothesis/vendor/tlds-alpha-by-domain.txt')

While using riot, the output is:

<importlib.resources._adapters.CompatibilityFiles.SpecPath object at 0x104befe50>
<importlib.resources._adapters.CompatibilityFiles.OrphanPath object at 0x104beeb50>

Currently unsure why using riot leads to a completely separate Path object.