Survey of installed Python dependencies (requirements.txt, commands.json, tox.ini, etc.)

Similar to https://github.com/web-platform-tests/wpt/issues/28801, I've had a look at what Python dependencies we install, rather than vendoring them into tool/third_party/. These are the mechanisms I'm aware of:

requirements.txt and requirements_*.txt files
install fields in commands.json files (and requirements fields which can refer to requirements.txt files)
deps sections in tox.ini files (which often refer to requirements.txt files)
direct calls to pip install in scripts or CI steps

Here are the dependencies, starting with some special groups and then alphabetically...

marionette_driver + mozcrash + mozdebug + mozinstall + mozleak + moznetwork + mozprofile + mozrunner + mozversion

These are dependencies for Firefox required only by tools/wptrunner/requirements_firefox.txt. (Some others are used more broadly, see below.)

flake8 + pep8-naming + pycodestyle + pyflakes

Dependencies for Python coding style in tools/tox.ini, via tools/requirements_flake8.txt. Not installed by any ./wpt command.

mypy + mypy-extensions + typed-ast

Dependencies for Python type checking in tools/tox.ini, via tools/requirements_mypy.txt. Not installed by any ./wpt command.

pytest + pytest-cov + hypothesis + mock

Dependencies for tests for tools/ in tools/tox.ini, via tools/requirements_pytest.txt. These are not the same as the pytest + dependencies used to run webdriver/tests/, those are separately vendored with different versions, see https://github.com/web-platform-tests/wpt/issues/28801.

requests + taskcluster + pyyaml + json-e + jsonschema

Dependencies in tools/tox.ini used in CI. Not installed by any ./wpt command. These aren't pinned, but I haven't seen things suddenly break because of them.

Sphinx + sphinx-argparse + recommonmark

These are in docs/requirements.txt, dependencies for building web-platform-tests.org, wired up to ./wpt build-docs but in CI that isn't used.

Template-Python + html5lib + lxml + mercurial + webencodings

These are the dependencies for css/build-css-testsuites.sh via css/requirements.txt. Seemingly not run in CI?

aioquic

Dependency for ./wpt serve-quic-transport in tools/quic/requirements.txt, but this is also a vendored dependency, see https://github.com/web-platform-tests/wpt/issues/28801. The versions aren't the same, which one ends up being used?

html5lib

In tools/wptrunner/requirements.txt, but is also a vendored dependency, see https://github.com/web-platform-tests/wpt/issues/28801. Notably, the webencodings dependency isn't in tools/wptrunner/requirements.txt...

Also in css/requirements.txt, see above.

Also in resources/test/tox.ini, used to extract inline test expectations.

mozinfo

In tools/wpt/requirements.txt and tools/wptrunner/requirements.txt, but seems like it's ultimately only used for Firefox, since it's related to mozinfo.json files that are probably only for Firefox.

mozinstall

Required by ./wpt install, via tools/wpt/commands.json.

mozlog

In tools/wptrunner/requirements.txt, used for logging in lots of places.

mozprocess

This is listed in many files:

All the usage is with mozprocess.ProcessHandler, in these files:

tools/wptrunner/wptrunner/browsers/chrome_android.py tools/wptrunner/wptrunner/browsers/firefox.py tools/wptrunner/wptrunner/browsers/servodriver.py tools/wptrunner/wptrunner/executors/executorservo.py tools/wptrunner/wptrunner/webdriver_server.py

Maybe this can be simplified somehow?

pillow

A dependency in tools/wptrunner/requirements.txt needed for RefTestImplementation.

psutil

From tools/wptrunner/requirements_safari.txt, used for killing Safari processes.

Also in tools/wptrunner/requirements_firefox.txt but seemingly not used for running Firefox?

requests

In these files: tools/wpt/requirements.txt tools/wptrunner/requirements.txt tools/wptrunner/requirements_sauce.txt

Used for requests.get and requests.post as you'd expect.

Also in resources/test/tox.ini and tools/tox.ini, but see https://github.com/web-platform-tests/wpt/pull/28777.

Also in tools/ci/commands.json, tools/docker/commands.json and tools/wpt/commands.json.

selenium

A dependency of anything built on tools/wptrunner/wptrunner/executors/executorselenium.py:

tools/wptrunner/requirements_edge.txt tools/wptrunner/requirements_ie.txt tools/wptrunner/requirements_opera.txt tools/wptrunner/requirements_sauce.txt

six

In tools/wptrunner/requirements.txt, https://github.com/web-platform-tests/wpt/issues/28776 tracks getting rid of wpt-internal usage. (It would still reamain as a transitive in dependency.)

Also in css/requirements.txt, see above.

Also in resources/test/tox.ini, but see https://github.com/web-platform-tests/wpt/pull/28777.

ua-parser

The only dependency in tools/wave/requirements.txt, used in tools/wave/utils/user_agent_parser.py.

urllib3

Listed in tools/wptrunner/requirements.txt, it's a dependency of selenium and requests, never used directly in wpt.

virtualenv

The only dependency for the ./wpt CLI beyond Python itself. Installed in CI, but no specific version is required. https://github.com/web-platform-tests/wpt/issues/20019 discusses avoiding this dependency.

wptserve

Listed in annotation-protocol/requirements.txt and can presumably be removed since wptserve is part of wpt now, see https://github.com/web-platform-tests/wpt/pull/28810.

zstandard

Required by ./wpt run, via tools/wpt/commands.json.

web-platform-tests / wpt