search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.37k stars 466 forks source link

System package information, spkg-configure for Jupyter "notebook" package, "rst2ipynb", and dependencies #30124

Open mkoeppe opened 4 years ago

mkoeppe commented 4 years ago

(From dimpase https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/4eKjmINXAQAJ)

We currently cannot use Python packages from the system (see #29023) - and spkg-configure.m4 is not the right tool for changing that.

However, the situation is different for the Jupyter notebook package (and its dependencies):

We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).

Likewise, rst2ipynb is a Python package but we only provide it to make a script available, not a Python module. So again, if the system provides a suitable version, we do not need to install it in our venv.

notebook's exclusive dependencies (see #26919, https://github.com/jupyter/notebook/blob/master/setup.py):

Shared between notebook and optional package rst2ipynb (via nbconvert):

The following dependencies are shared between notebook and sagelib/ipython/ipykernel. So we need to install them even if they are present in the system python:

Packages depending on notebook:

See also:

Depends on #28197 Depends on #30299

CC: @dimpase @nbruin @slel @paulmasson @kiwifb @EmmanuelCharpentier @jhpalmieri @thierry-FreeBSD @sheerluck @nthiery

Component: build: configure

Keywords: sd111

Branch/Commit: u/mkoeppe/system_information__spkg_configure_for_jupyternotebookpackage_and_dependencies @ b8164c1

Issue created by migration from https://trac.sagemath.org/ticket/30124

mkoeppe commented 4 years ago
comment:2

This code from src/sage/repl/ipython_kernel/install.py of course needs updating so that we can install the kernel outside of the Sage venv:

        if prefix is None:
            from sys import prefix
        jupyter_dir = os.path.join(prefix, "share", "jupyter")
        self.nbextensions_dir = os.path.join(jupyter_dir, "nbextensions")
        self.kernel_dir = os.path.join(jupyter_dir, "kernels", self.identifier())
        self._mkdirs()
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -4,4 +4,8 @@
 Sage to supply anything but the jupyter kernel...

+See also:
+- #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
+- https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion

+
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,11 +1,13 @@
-From dimpase https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/4eKjmINXAQAJ:
+(From dimpase https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/4eKjmINXAQAJ)

-switch to external jupyter - there is  no reason for
-Sage to supply anything but the jupyter kernel...
+We currently cannot use Python packages from the system (see #29023) - and `spkg-configure.m4` is not the right tool for changing that.
+
+However, the situation is different for the Jupyter `notebook` package (and its dependencies): 
+
+We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).
+

 See also:
 - #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
-- https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion

-
kiwifb commented 4 years ago
comment:6

That implies that we will have to ask a jupyter that is in the PATH to tell us about its installation PATH.

If we just call jupyter kernelspec install then it will install in a user location for a system wide jupyter (~/.local/share/jupyter/kernels on linux), for a sage jupyter it will probably go to the sage install ({sys.prefix}/share/jupyter/kernels on linux/OS X). But for distro it will become problematic straight away since you install in a stage before merging in the system tree. It is possible to pass a --prefix but this is not the same as DESTDIR which would be the useful thing for distros.

mkoeppe commented 4 years ago
comment:7

I think we will only need the following changes to sagelib:

Calling sage-kernel would be done by sage-the-distribution (perhaps in build/pkgs/sagelib/spkg-install), and likewise by distribution packagers' scripts.

mkoeppe commented 4 years ago
comment:8

Replying to @mkoeppe:

I think we will only need the following changes to sagelib: ... Calling sage-kernel would be done by sage-the-distribution (perhaps in build/pkgs/sagelib/spkg-install)

I'll do the first two items in #30298

mkoeppe commented 4 years ago
comment:9

I have changed the plan a bit, see #30298 and #30299.

mkoeppe commented 4 years ago

Dependencies: #28197, #30299

mkoeppe commented 4 years ago

Branch: u/mkoeppe/system_information__spkg_configure_for_jupyternotebookpackage_and_dependencies

mkoeppe commented 4 years ago

Commit: b8164c1

mkoeppe commented 4 years ago

New commits:

b8164c1build/pkgs/notebook/spkg-configure.m4: New
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -10,4 +10,4 @@

 See also:
 - #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
-
+- Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -6,8 +6,18 @@

 We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).

+`notebook`'s dependencies (see #26919):
+- `nbformat`, `nbconvert`, `terminado`, `prometheus_client`, `send2trash`
+
+The following dependencies are shared between `notebook` and `sagelib`/`ipython`/`ipykernel`. So we need to install them even if they are present in the system python:
+- `pyzmq`, `dateutil`, `tornado`, jupyter-client`, `jupyter-core`, `jinja2`
+
+Packages depending on `notebook`:
+- `pari_jupyter` (no system packages: https://repology.org/projects/?search=pari_jupyter&maintainer=&category=&inrepo=&notinrepo=&repos=&families=&repos_newest=&families_newest=)
+- `r_jupyter` (script package; https://repology.org/project/r:irkernel/versions)
+- `widgetsnbextension`

 See also:
 - #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
-- Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
+- #26919 Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -6,11 +6,15 @@

 We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).

-`notebook`'s dependencies (see #26919):
-- `nbformat`, `nbconvert`, `terminado`, `prometheus_client`, `send2trash`
+`notebook`'s exclusive dependencies (see #26919):
+- `terminado`, `prometheus_client`, `send2trash`
+
+Shared between `notebook` and optional package `rst2ipynb` (via `nbconvert`):
+- `nbconvert`, `nbformat`, `jsonschema`, `mistune`, `entrypoints`, `bleach`, `pandocfilters`, `testpath`, `defusedxml`
+- `pandoc`, `pandoc_attributes` (?)

 The following dependencies are shared between `notebook` and `sagelib`/`ipython`/`ipykernel`. So we need to install them even if they are present in the system python:
-- `pyzmq`, `dateutil`, `tornado`, jupyter-client`, `jupyter-core`, `jinja2`
+- `pyzmq`, `dateutil`, `tornado`, `jupyter-client`, `jupyter-core`, `jinja2`, `pygments`

 Packages depending on `notebook`:
 - `pari_jupyter` (no system packages: https://repology.org/projects/?search=pari_jupyter&maintainer=&category=&inrepo=&notinrepo=&repos=&families=&repos_newest=&families_newest=)
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -6,15 +6,17 @@

 We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).

-`notebook`'s exclusive dependencies (see #26919):
+`notebook`'s exclusive dependencies (see #26919, https://github.com/jupyter/notebook/blob/master/setup.py):
 - `terminado`, `prometheus_client`, `send2trash`
+- `argon2-cffi` (new dependency)

 Shared between `notebook` and optional package `rst2ipynb` (via `nbconvert`):
 - `nbconvert`, `nbformat`, `jsonschema`, `mistune`, `entrypoints`, `bleach`, `pandocfilters`, `testpath`, `defusedxml`
+- `pyrsistent` (new dependency of `jsonschema`)
 - `pandoc`, `pandoc_attributes` (?)

 The following dependencies are shared between `notebook` and `sagelib`/`ipython`/`ipykernel`. So we need to install them even if they are present in the system python:
-- `pyzmq`, `dateutil`, `tornado`, `jupyter-client`, `jupyter-core`, `jinja2`, `pygments`
+- `pyzmq`, `dateutil`, `tornado`, `jupyter-client`, `jupyter-core`, `jinja2`, `pygments`, `ipython_genutils`

 Packages depending on `notebook`:
 - `pari_jupyter` (no system packages: https://repology.org/projects/?search=pari_jupyter&maintainer=&category=&inrepo=&notinrepo=&repos=&families=&repos_newest=&families_newest=)
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -5,6 +5,8 @@
 However, the situation is different for the Jupyter `notebook` package (and its dependencies): 

 We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).
+
+Likewise, `rst2ipynb` is a Python package but we only provide it to make a script available, not a Python module. So again, if the system provides a suitable version, we do not need to install it in our venv.

 `notebook`'s exclusive dependencies (see #26919, https://github.com/jupyter/notebook/blob/master/setup.py):
 - `terminado`, `prometheus_client`, `send2trash`
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -29,3 +29,4 @@
 See also:
 - #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
 - #26919 Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
+- #30246 Support interacts in JupyterLab
mkoeppe commented 3 years ago
comment:20

If we attempt to automatically install the Sage jupyter kernel into the system jupyter, for example using jupyter kernelspec install --user as explained in #30476, we should think about how to disambiguate the kernels when working with multiple Sage installations.

mkoeppe commented 3 years ago
comment:21

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

mkoeppe commented 3 years ago

Changed keywords from none to sd111

mkoeppe commented 3 years ago
comment:22

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.