pythonPackages.selenium: FileNotFoundError: [Errno 2] No such file or directory, 'getAttribute.js'

polykernel commented 2 years ago

Describe the bug

The selenium python bindings includes additional build artifacts(i.e javascript file) in its build process which are depended on for certain functionality. However, these files are not included when building with setuptools as they are built by bazel and copied into the source. This result in errors at runtime when certain functions are called which requires loading the javascript files. For instance, get_attribute and is_displayed.

This seems to be caused by the version bump 3.141.0 -> 4.2.0 in https://github.com/NixOS/nixpkgs/commit/8466ac2820966da53ff0d8aafbb01380cab195e4 as prior versions are built using the sdist tarball on PyPi which include the necessary build artifacts. Version 4.0.0 do not have corresponding sdist tarball available as reported in https://github.com/SeleniumHQ/selenium/issues/9917 and subsequent version also do not seem to have them.

Relevant files: https://github.com/SeleniumHQ/selenium/blob/7b1c6461f618fefcb8fec4a972bc7ec1abb47c34/py/BUILD.bazel#L84-L107

Steps To Reproduce

Rough reproducible example:

  #!/usr/bin/env nix-shell
  #!nix-shell -i python3 -p "python39.withPackages(ps: [ ps.selenium ])"
  #!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/87e7965bbcdbac3d103e3ed14ff04f719a4f7a58.tar.gz
  from selenium import webdriver
  from selenium.webdriver.firefox.options import Options
  from selenium.webdriver.common.by import By

  options = Options()
  options.headless = True

  driver = webdriver.Firefox(options=options)
  driver.get("https://quotes.toscrape.com/")
  element = driver.find_element(By.XPATH, "//div[@class='quote']")
  print(element.get_attribute("innerHTML"))
  driver.quit()

Error log

``` Traceback (most recent call last): File "/home/user/test.py", line 11, in print(element.get_attribute("innerHTML")) File "/nix/store/nlcjwyijydldr02ynhsa9kv5wvxs44b3-python3-3.9.13-env/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 172, in get_attribute _load_js() File "/nix/store/nlcjwyijydldr02ynhsa9kv5wvxs44b3-python3-3.9.13-env/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 45, in _load_js getAttribute_js = pkgutil.get_data(_pkg, 'getAttribute.js').decode('utf8') File "/nix/store/r8hi2lpzycparplcq0y663c43866f9bz-python3-3.9.13/lib/python3.9/pkgutil.py", line 639, in get_data return loader.get_data(resource_name) File "", line 1039, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/nix/store/nlcjwyijydldr02ynhsa9kv5wvxs44b3-python3-3.9.13-env/lib/python3.9/site-packages/selenium/webdriver/remote/getAttribute.js' ```

Expected behavior

The javascript build artifacts are present in the python environment and the get_attribute function call completes successfully.

Additional context

According to https://github.com/SeleniumHQ/selenium/issues/10083#issuecomment-982068916, the recommended way to build from source is invoking bazel build //py:selenium-wheel on the source repository. I am not familiar with bazel and the python packaging framework in Nixpkgs but would it be possible to utilize bazel in the selenium build process?

Notify maintainers

@jraygauthier @SuperSandro2000

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.18.9, NixOS, 22.11 (Raccoon), 22.11.20220708.b39924f`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.9.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

SuperSandro2000 commented 2 years ago

According to SeleniumHQ/selenium#10083 (comment), the recommended way to build from source is invoking bazel build //py:selenium-wheel on the source repository. I am not familiar with bazel and the python packaging framework in Nixpkgs but would it be possible to utilize bazel in the selenium build process?

bazel is always a huge pain point and makes everything in nix a bit harder than necessary. So I would really try to avoid using it. We probably end up doing the build steps manually.

SuperSandro2000 commented 2 years ago

Couldn't figure out what bazel is doing behind its multiple layers of abstraction. Only option I see is to fetch the wheel which is also ugly.

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-fix-selenium-python-package-locally/23660/1

paulhersch commented 1 year ago

I have the same issue, any progress on a fix?

~~EDIT: quick workaround currently being that i use python packages from 22.05, though this is an unacceptable state~~ they are also broken now

reflektoin commented 1 year ago

I haven't tried this yet myself, but the NixOS Discourse mentioned earlier has a workaround: https://discourse.nixos.org/t/how-to-fix-selenium-python-package-locally/23660/4

NixOS / nixpkgs