ModuleNotFoundError: No module named 'selenium.webdriver.common.devtools'FileNotFoundError: [Errno 2] No such file or directory: '/nix/store/j5xnifvfvbsf66qj50v9ns355xfv3hsr-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/devtools'
selenium.nix
[nixpkgs/pkgs/development/python-modules/selenium/default.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/selenium/default.nix)
```nix
buildPythonPackage rec {
pname = "selenium";
version = "4.14.0";
format = "setuptools";
disabled = pythonOlder "3.7";
src = fetchFromGitHub {
owner = "SeleniumHQ";
repo = "selenium";
# check if there is a newer tag with or without -python suffix
rev = "refs/tags/selenium-${version}";
hash = "sha256-cTMCKfFLUlJDbTUQA3Z/pKCE1RQQRMb4K8hKKn9HqvU=";
};
```
upstream uses the gn build system to fix the issue, but the pypi "Source Distribution" is still broken
the pypi "Built Distribution" works, because it has the selenium/webdriver/common/devtools/*/*.py files
fix 1
in setup.py use find_packages
and add an empty selenium/webdriver/common/devtools/__init__.py file
fix 2
in setup.py use find_namespace_packages
because there is no selenium/webdriver/common/devtools/__init__.py
When you use find_packages(), all directories without an __init__.py file will be ignored. On the other hand, find_namespace_packages() will scan all directories.
setuptools/command/build_py.py:207: _Warning: Package 'selenium.webdriver.common.devtools.v119' is absent from the `packages` configuration.
```
$ nix log /nix/store/23rs2vj83drvqw0fy4631kp11f4v30ma-python3.11-selenium-4.17.2
!!
check.warn(importable)
/nix/store/7fnhc3nd125lq9vzw6bghsvbnmcc8mc1-python3.11-setuptools-69.0.2/lib/python3.11/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'selenium.webdriver.common.devtools.v119' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'selenium.webdriver.common.devtools.v119' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'selenium.webdriver.common.devtools.v119' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'selenium.webdriver.common.devtools.v119' to be distributed and are
already explicitly excluding 'selenium.webdriver.common.devtools.v119' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
```
what packages are missing
```
$ nix log /nix/store/23rs2vj83drvqw0fy4631kp11f4v30ma-python3.11-selenium-4.17.2 | grep "Python recognizes"
Python recognizes 'selenium.webdriver.common.actions' as an importable package[^1],
Python recognizes 'selenium.webdriver.common.bidi' as an importable package[^1],
Python recognizes 'selenium.webdriver.common.devtools.v119' as an importable package[^1],
Python recognizes 'selenium.webdriver.common.devtools.v120' as an importable package[^1],
Python recognizes 'selenium.webdriver.common.devtools.v121' as an importable package[^1],
Python recognizes 'selenium.webdriver.common.devtools.v85' as an importable package[^1],
Python recognizes 'selenium.webdriver.safari' as an importable package[^1],
Python recognizes 'selenium.webdriver.webkitgtk' as an importable package[^1],
Python recognizes 'selenium.webdriver.wpewebkit' as an importable package[^1],
```
setup.py
```py
'packages': ['selenium',
'selenium.common',
'selenium.webdriver',
'selenium.webdriver.chromium',
'selenium.webdriver.chrome',
'selenium.webdriver.common',
'selenium.webdriver.support',
'selenium.webdriver.firefox',
'selenium.webdriver.ie',
'selenium.webdriver.edge',
'selenium.webdriver.remote',
'selenium.webdriver.support', ],
'include_package_data': True,
```
repro.py
`repro.py` from https://stackoverflow.com/questions/66227508
```py
#!/usr/bin/env python3
import trio # async library that selenium uses
from selenium import webdriver
async def start_listening(listener):
async for event in listener:
print(event)
async def main():
driver = webdriver.Chrome()
async with driver.bidi_connection() as connection:
session, devtools = connection.session, connection.devtools
# await session.execute(devtools.fetch.enable())
await session.execute(devtools.network.enable())
# listener = session.listen(devtools.fetch.RequestPaused)
listener = session.listen(devtools.network.ResponseReceived)
async with trio.open_nursery() as nursery:
nursery.start_soon(start_listening, listener) # start_listening blocks, so we run it in another coroutine
url = "https://httpbin.org/get" # json response
driver.get(url)
trio.run(main)
```
simply using the pypi source does not work: the selenium-manager binary is missing at /lib/python3.11/site-packages/selenium/webdriver/common/linux/selenium-manager
```nix
{ lib
, fetchPypi
, buildPythonPackage
, setuptools
, wheel
, certifi
, trio
, trio-websocket
, typing-extensions
, urllib3
}:
buildPythonPackage rec {
pname = "selenium";
version = "4.17.2";
pyproject = true;
src = fetchPypi {
inherit pname version;
hash = "sha256-1D1pcuUWhV+yQu+c5M51kFexFQcOcC57HBAy/ns42Hs=";
};
# relax versions
# fix: typing-extensions~=4.9 not satisfied by version 4.8.0
# https://github.com/SeleniumHQ/selenium/blob/trunk/py/setup.py
# also add selenium.webdriver.common.devtools.* packages
# fix: ModuleNotFoundError: No module named 'selenium.webdriver.common.devtools'
# https://github.com/milahu/nixpkgs/issues/20
postPatch = ''
sed -i 's/[~>]=.*"/"/' setup.py
substituteInPlace setup.py \
--replace \
" 'packages': [" \
" 'packages': find_namespace_packages(where='.'), '_packages': [" \
--replace \
"from setuptools import setup" \
"from setuptools import setup, find_namespace_packages" \
'';
nativeBuildInputs = [
setuptools
wheel
];
propagatedBuildInputs = [
certifi
trio
trio-websocket
typing-extensions
urllib3
];
pythonImportsCheck = [ "selenium" ];
meta = with lib; {
description = "Bindings for Selenium WebDriver";
homepage = "https://selenium.dev/";
license = licenses.asl20;
maintainers = with maintainers; [ ];
};
}
```
```
$ ./repro.py
Traceback (most recent call last):
File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/driver_finder.py", line 38, in get_path
path = SeleniumManager().driver_location(options) if path is None else path
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/selenium_manager.py", line 89, in driver_location
args = [str(self.get_binary()), "--browser", browser]
^^^^^^^^^^^^^^^^^
File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/selenium_manager.py", line 73, in get_binary
raise WebDriverException(f"Unable to obtain working Selenium Manager binary; {path}")
selenium.common.exceptions.WebDriverException: Message: Unable to obtain working Selenium Manager binary; /nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/linux/selenium-manager
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/src/milahu/opensubtitles-scraper/./aiohttp_chromium/test/stream-response/Network.dataReceived.selenium.2.py", line 38, in
trio.run(main)
File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/trio/_core/_run.py", line 2251, in run
raise runner.main_task_outcome.error
File "/home/user/src/milahu/opensubtitles-scraper/./aiohttp_chromium/test/stream-response/Network.dataReceived.selenium.2.py", line 14, in main
driver = webdriver.Chrome()
^^^^^^^^^^^^^^^^^^
File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
super().__init__(
File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/chromium/webdriver.py", line 49, in __init__
self.service.path = DriverFinder.get_path(self.service, options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/driver_finder.py", line 41, in get_path
raise NoSuchDriverException(msg) from err
selenium.common.exceptions.NoSuchDriverException: Message: Unable to obtain driver for chrome using Selenium Manager.; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/driver_location
```
problem:
python3Packages.seleniumis built from github source, not from pypi sourceselenium.nix
[nixpkgs/pkgs/development/python-modules/selenium/default.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/selenium/default.nix) ```nix buildPythonPackage rec { pname = "selenium"; version = "4.14.0"; format = "setuptools"; disabled = pythonOlder "3.7"; src = fetchFromGitHub { owner = "SeleniumHQ"; repo = "selenium"; # check if there is a newer tag with or without -python suffix rev = "refs/tags/selenium-${version}"; hash = "sha256-cTMCKfFLUlJDbTUQA3Z/pKCE1RQQRMb4K8hKKn9HqvU="; }; ```upstream issue: https://github.com/SeleniumHQ/selenium/issues/9010
upstream uses the
gnbuild system to fix the issue, but the pypi "Source Distribution" is still broken the pypi "Built Distribution" works, because it has theselenium/webdriver/common/devtools/*/*.pyfilesfix 1
in
setup.pyusefind_packagesand add an emptyselenium/webdriver/common/devtools/__init__.pyfilefix 2
in
setup.pyuse find_namespace_packages because there is noselenium/webdriver/common/devtools/__init__.pydone in selenium.nix
fix 3
use the pypi "Built Distribution" this works, because it has the
selenium/webdriver/common/devtools/*/*.pyfilesfix 4
fix the
buildPhaseto produce the missing files see the upstream issue: https://github.com/SeleniumHQ/selenium/issues/9010setuptools/command/build_py.py:207: _Warning: Package 'selenium.webdriver.common.devtools.v119' is absent from the `packages` configuration.
``` $ nix log /nix/store/23rs2vj83drvqw0fy4631kp11f4v30ma-python3.11-selenium-4.17.2 !! check.warn(importable) /nix/store/7fnhc3nd125lq9vzw6bghsvbnmcc8mc1-python3.11-setuptools-69.0.2/lib/python3.11/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'selenium.webdriver.common.devtools.v119' is absent from the `packages` configuration. !! ******************************************************************************** ############################ # Package would be ignored # ############################ Python recognizes 'selenium.webdriver.common.devtools.v119' as an importable package[^1], but it is absent from setuptools' `packages` configuration. This leads to an ambiguous overall configuration. If you want to distribute this package, please make sure that 'selenium.webdriver.common.devtools.v119' is explicitly added to the `packages` configuration field. Alternatively, you can also rely on setuptools' discovery methods (for example by using `find_namespace_packages(...)`/`find_namespace:` instead of `find_packages(...)`/`find:`). You can read more about "package discovery" on setuptools documentation page: - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html If you don't want 'selenium.webdriver.common.devtools.v119' to be distributed and are already explicitly excluding 'selenium.webdriver.common.devtools.v119' via `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`, you can try to use `exclude_package_data`, or `include-package-data=False` in combination with a more fine grained `package-data` configuration. You can read more about "package data files" on setuptools documentation page: - https://setuptools.pypa.io/en/latest/userguide/datafiles.html [^1]: For Python, any directory (with suitable naming) can be imported, even if it does not contain any `.py` files. On the other hand, currently there is no concept of package data directory, all directories are treated like packages. ******************************************************************************** ```what packages are missing
``` $ nix log /nix/store/23rs2vj83drvqw0fy4631kp11f4v30ma-python3.11-selenium-4.17.2 | grep "Python recognizes" Python recognizes 'selenium.webdriver.common.actions' as an importable package[^1], Python recognizes 'selenium.webdriver.common.bidi' as an importable package[^1], Python recognizes 'selenium.webdriver.common.devtools.v119' as an importable package[^1], Python recognizes 'selenium.webdriver.common.devtools.v120' as an importable package[^1], Python recognizes 'selenium.webdriver.common.devtools.v121' as an importable package[^1], Python recognizes 'selenium.webdriver.common.devtools.v85' as an importable package[^1], Python recognizes 'selenium.webdriver.safari' as an importable package[^1], Python recognizes 'selenium.webdriver.webkitgtk' as an importable package[^1], Python recognizes 'selenium.webdriver.wpewebkit' as an importable package[^1], ```setup.py
```py 'packages': ['selenium', 'selenium.common', 'selenium.webdriver', 'selenium.webdriver.chromium', 'selenium.webdriver.chrome', 'selenium.webdriver.common', 'selenium.webdriver.support', 'selenium.webdriver.firefox', 'selenium.webdriver.ie', 'selenium.webdriver.edge', 'selenium.webdriver.remote', 'selenium.webdriver.support', ], 'include_package_data': True, ```repro.py
`repro.py` from https://stackoverflow.com/questions/66227508 ```py #!/usr/bin/env python3 import trio # async library that selenium uses from selenium import webdriver async def start_listening(listener): async for event in listener: print(event) async def main(): driver = webdriver.Chrome() async with driver.bidi_connection() as connection: session, devtools = connection.session, connection.devtools # await session.execute(devtools.fetch.enable()) await session.execute(devtools.network.enable()) # listener = session.listen(devtools.fetch.RequestPaused) listener = session.listen(devtools.network.ResponseReceived) async with trio.open_nursery() as nursery: nursery.start_soon(start_listening, listener) # start_listening blocks, so we run it in another coroutine url = "https://httpbin.org/get" # json response driver.get(url) trio.run(main) ```simply using the pypi source does not work: the selenium-manager binary is missing at /lib/python3.11/site-packages/selenium/webdriver/common/linux/selenium-manager
```nix { lib , fetchPypi , buildPythonPackage , setuptools , wheel , certifi , trio , trio-websocket , typing-extensions , urllib3 }: buildPythonPackage rec { pname = "selenium"; version = "4.17.2"; pyproject = true; src = fetchPypi { inherit pname version; hash = "sha256-1D1pcuUWhV+yQu+c5M51kFexFQcOcC57HBAy/ns42Hs="; }; # relax versions # fix: typing-extensions~=4.9 not satisfied by version 4.8.0 # https://github.com/SeleniumHQ/selenium/blob/trunk/py/setup.py # also add selenium.webdriver.common.devtools.* packages # fix: ModuleNotFoundError: No module named 'selenium.webdriver.common.devtools' # https://github.com/milahu/nixpkgs/issues/20 postPatch = '' sed -i 's/[~>]=.*"/"/' setup.py substituteInPlace setup.py \ --replace \ " 'packages': [" \ " 'packages': find_namespace_packages(where='.'), '_packages': [" \ --replace \ "from setuptools import setup" \ "from setuptools import setup, find_namespace_packages" \ ''; nativeBuildInputs = [ setuptools wheel ]; propagatedBuildInputs = [ certifi trio trio-websocket typing-extensions urllib3 ]; pythonImportsCheck = [ "selenium" ]; meta = with lib; { description = "Bindings for Selenium WebDriver"; homepage = "https://selenium.dev/"; license = licenses.asl20; maintainers = with maintainers; [ ]; }; } ``` ``` $ ./repro.py Traceback (most recent call last): File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/driver_finder.py", line 38, in get_path path = SeleniumManager().driver_location(options) if path is None else path ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/selenium_manager.py", line 89, in driver_location args = [str(self.get_binary()), "--browser", browser] ^^^^^^^^^^^^^^^^^ File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/selenium_manager.py", line 73, in get_binary raise WebDriverException(f"Unable to obtain working Selenium Manager binary; {path}") selenium.common.exceptions.WebDriverException: Message: Unable to obtain working Selenium Manager binary; /nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/linux/selenium-manager The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/src/milahu/opensubtitles-scraper/./aiohttp_chromium/test/stream-response/Network.dataReceived.selenium.2.py", line 38, insimilar issues
ping
ping maintainer @jraygauthier
ping contributors @arkivm @sdht0 @AleXoundOS @mweinelt @contrun @SuperSandro2000 @lapp0 @polykernel @paulhersch @reflektoin