Standard wheel packages

[mkoeppe]

We add a mechanism to install Python packages from (platform-independent) wheels instead of from source distributions.

This will allow us to make packages such as furo and jupyterlab standard. Currently they are pip packages, as they need internet connectivity to get Javascript packages (via nodeenv).

Here we make furo a standard wheel package. This is for #33833.

Depends on #34421

CC: @dimpase @kwankyu @jhpalmieri

Component: build

Author: Matthias Koeppe

Branch/Commit: d887d44

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/34450

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Branch: u/mkoeppe/standard_wheel_packages

[mkoeppe]

Description changed:

--- 
+++ 
@@ -2,3 +2,4 @@

 This will allow us to make packages such as `furo` and `jupyterlab` standard. Currently they are `pip` packages, as they need internet connectivity to get Javascript packages (via `nodeenv`).

+Here we make `furo` a standard wheel package.

[mkoeppe]

Commit: fd1c677

[mkoeppe]

New commits:

8d90b56	build/pkgs/furo: Make it a normal package
28f30ac	build/bin/sage-spkg: Handle *.whl packages
fd1c677	build/pkgs/furo: Use a wheel

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from fd1c677 to 9fa490a

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9fa490a

build/pkgs/sphinx_basic_ng: New (furo dependency)

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

eba7f6f

src/doc/en/developer/packaging.rst: Document wheel packages

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9fa490a to eba7f6f

[mkoeppe]

Description changed:

--- 
+++ 
@@ -2,4 +2,4 @@

 This will allow us to make packages such as `furo` and `jupyterlab` standard. Currently they are `pip` packages, as they need internet connectivity to get Javascript packages (via `nodeenv`).

-Here we make `furo` a standard wheel package.
+Here we make `furo` a standard wheel package. This is for #33833.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

8e06d43

src/doc/en/developer/packaging.rst: Policy on wheels, on standard packages

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from eba7f6f to 8e06d43

[kwankyu]

comment:8

If non-platform is not already a standard term, then I suggest platform-independent, which is longer but easier to understand and reads well than non-platform

[mkoeppe]

comment:9

Please feel free to push improvements to the documentation to the ticket. I have to stop here for tonight.

I don't see "platform-independent" in https://peps.python.org/pep-0425/; it uses the terminology "pure Python". There might be other places to look at

[kwankyu]

comment:10

This failure seems related:

Installing jupyter_jsmol-2022.1.0
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Looking in links: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels
Processing /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Looking in links: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels
  ERROR: Could not find a version that satisfies the requirement jupyter_packaging~=0.7.9 (from versions: none)
  ERROR: No matching distribution found for jupyter_packaging~=0.7.9
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin/python3 /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-standalone-pip-lw2mb0lw/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-build-env-l0gj_p8c/overlay --no-warn-script-location --no-binary :all: --only-binary :none: --no-index --find-links /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels -- 'jupyter_packaging~=0.7.9' 'jupyterlab>=3.0.0,==3.*' 'setuptools>=40.8.0' wheel
  cwd: [inherit]
  Installing build dependencies: finished with status 'error'
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Warning: building with "python3 -m pip wheel --wheel-dir=dist --verbose --no-deps --no-index --isolated --ignore-requires-python --find-links=/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels" failed.
Retrying with "python3 -m pip wheel --wheel-dir=dist --verbose --no-deps --no-index --isolated --ignore-requires-python --no-build-isolation --no-binary :all:".
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Processing /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  setup.py entered
  $PATH=/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/src/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/usr/local/opt/polymake/bin:/usr/local/opt/texinfo/bin:/usr/local/opt/gettext/bin:/Applications/google-cloud-sdk/bin:.:/Users/kwankyu/Workplace/bin:/usr/local/bin:/usr/local/sbin:/usr/texbin:/Applications/Macaulay2-1.20/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/Library/Apple/usr/bin:/Applications/Magma
  <string>:47: DeprecatedWarning: create_cmdclass is deprecated as of 0.8 and will be removed in 1.0. "
  Use `wrap_installers` to handle prebuild steps in cmdclass.
  Use `get_data_files` to handle data files.
  Use `include_package_data=True` and `MANIFEST.in` for package data.

  <string>:49: DeprecatedWarning: install_npm is deprecated as of 0.8 and will be removed in 1.0. Use `npm_builder` and `wrap_installers`
  running dist_info
  creating /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info
  writing /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/dependency_links.txt
  writing requirements to /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/requires.txt
  writing top-level names to /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/top_level.txt
  writing manifest file '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'examples'
  no previously-included directories found matching 'tests/build'
  no previously-included directories found matching '**/node_modules'
  warning: no previously-included files matching '*~' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.pyo' found anywhere in distribution
  warning: no previously-included files matching '.git' found anywhere in distribution
  warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
  warning: no previously-included files matching '*.map' found anywhere in distribution
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/SOURCES.txt'
  creating '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol-2022.1.0.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: jupyter-jsmol
  WARNING: Ignoring --build-option when building jupyter-jsmol using PEP 517
  Building wheel for jupyter-jsmol (pyproject.toml): started
  Running command Building wheel for jupyter-jsmol (pyproject.toml)
  setup.py entered
  $PATH=/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/src/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/usr/local/opt/polymake/bin:/usr/local/opt/texinfo/bin:/usr/local/opt/gettext/bin:/Applications/google-cloud-sdk/bin:.:/Users/kwankyu/Workplace/bin:/usr/local/bin:/usr/local/sbin:/usr/texbin:/Applications/Macaulay2-1.20/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/Library/Apple/usr/bin:/Applications/Magma
  <string>:47: DeprecatedWarning: create_cmdclass is deprecated as of 0.8 and will be removed in 1.0. "
  Use `wrap_installers` to handle prebuild steps in cmdclass.
  Use `get_data_files` to handle data files.
  Use `include_package_data=True` and `MANIFEST.in` for package data.

  <string>:49: DeprecatedWarning: install_npm is deprecated as of 0.8 and will be removed in 1.0. Use `npm_builder` and `wrap_installers`
  running bdist_wheel
  running jsdeps
  Installing build dependencies with npm.  This may take a while...
  > yarn install
  yarn install v1.22.19
  warning package.json: No license field
  warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
  warning jupyter-jsmol@2022.1.0: No license field
  [1/4] Resolving packages...
  [2/4] Fetching packages...
  info There appears to be trouble with your network connection. Retrying...
  info There appears to be trouble with your network connection. Retrying...
  info There appears to be trouble with your network connection. Retrying...
  info There appears to be trouble with your network connection. Retrying...
  error An unexpected error occurred: "https://registry.yarnpkg.com/@jupyterlab/nbformat/-/nbformat-3.1.18.tgz: tunneling socket could not be established, cause=connect ETIMEDOUT 192.0.2.0:5187".
  info If you think this is a bug, please open a bug report with the information provided in "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src/js/yarn-error.log".
  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
  Traceback (most recent call last):
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 244, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 229, in _build_with_temp_dir
      self.run_setup()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
      exec(code, locals())
    File "<string>", line 95, in <module>
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 177, in setup
      return run_commands(dist)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 193, in run_commands
      dist.run_commands()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
      self.run_command(cmd)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
      cmd_obj.run()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 725, in run
      [self.run_command(cmd) for cmd in cmds]
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 725, in <listcomp>
      [self.run_command(cmd) for cmd in cmds]
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 317, in run_command
      self.distribution.run_command(command)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
      cmd_obj.run()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 344, in run
      c.run()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 694, in run
      builder()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 230, in builder
      run(npm_cmd + ["install"], cwd=node_package)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 296, in run
      return subprocess.check_call(cmd, **kwargs)
    File "/usr/local/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/local/bin/yarn', 'install']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Building wheel for jupyter-jsmol (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin/python3 /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/tmp1q2r7ekk
  cwd: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src
  Building wheel for jupyter-jsmol (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for jupyter-jsmol
Failed to build jupyter-jsmol
ERROR: Failed to build one or more wheels
********************************************************************************
Error building a wheel for jupyter_jsmol-2022.1.0
******************************************************************************** 

[kwankyu]

Changed branch from u/mkoeppe/standard_wheel_packages to u/klee/standard_wheel_packages

[kwankyu]

comment:12

Replying to @mkoeppe:

I don't see "platform-independent" in https://peps.python.org/pep-0425/; it uses the terminology "pure Python". There might be other places to look at

Just from internet search, I see no "non-platform" and a couple of "platform independent". As far as I understand, "pure Python" is not quite right.

---

New commits:

b12e9dc

Use the name platform-independent

[kwankyu]

Changed commit from 8e06d43 to b12e9dc

[dimpase]

comment:13

You need #34421, where this jsmol issue was fixed. (Volker just merged it, but it's not yet in a beta)

---

New commits:

b12e9dc

Use the name platform-independent

[dimpase]

Dependencies: #34421

[kwankyu]

comment:14

With #34421 and this ticket, #33833 works well!

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from b12e9dc to d887d44

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d887d44

Use rerun instead of re-run

[kwankyu]

comment:16

LGTM.

[kwankyu]

Reviewer: Kwankyu Lee

[mkoeppe]

comment:17

Thanks!

[kwankyu]

comment:19

Do you know why the github actions failed?

[kwankyu]

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-We add a mechanism to install Python packages from (non-platform) wheels instead of from source distributions.
+We add a mechanism to install Python packages from (platform-independent) wheels instead of from source distributions.

 This will allow us to make packages such as `furo` and `jupyterlab` standard. Currently they are `pip` packages, as they need internet connectivity to get Javascript packages (via `nodeenv`).

[mkoeppe]

comment:20

[sphinx_basic_ng-0.0.1a12] cat: /sage/build/pkgs/sphinx_basic_ng/type: No such file or directory

Maybe I forgot to check in a file?

[kwankyu]

comment:21

The file is there.

[mkoeppe]

comment:22

It's a defect of the type of incremental build with a changed SAGE_ROOT, specific to the GH Actions workflow. Nothing wrong with this ticket.

[mkoeppe]

comment:23

I've opened #34452 for it

[kwankyu]

comment:24

Replying to @mkoeppe:

I've opened #34452 for it

Thanks.

Sage development is like mathematics research: if you solved a problem, you see a new problem from it. It never ends :)

[vbraun]

Changed branch from u/klee/standard_wheel_packages to d887d44