Error with file patching for recent pygments/jupyter versions

[cameronraysmith]

Installing maxima-jupyter in https://github.com/cameronraysmith/notebooks/blob/32448d9a5ad0f0cb87ad0ee6c9b48d9d47baa1cd/Dockerfile#L72 against recent pygments/jupyter versions (see below)

jupyter installation log

``` Sun, 15 Aug 2021 17:21:58 GMT #12 [ 7/39] RUN pip install wheel jupyter jupyterlab git+jupyterlab/jupyterlab-git.git nbresuse jupyterlab-topbar jupyterlab-system-monitor jupytext RISE voila jupyterlab-sos aquirdturtle_collapsible_headings jupyterlab-execute-time jupyterlab-skip-traceback nbgitpuller jupyterhub && jupyter lab build #12 0.510 Collecting git+jupyterlab/jupyterlab-git.git #12 0.511 Cloning jupyterlab/jupyterlab-git.git to /tmp/pip-req-build-4pv6k6pz #12 0.511 Running command git clone -q jupyterlab/jupyterlab-git.git /tmp/pip-req-build-4pv6k6pz #12 2.168 Installing build dependencies: started #12 21.78 Installing build dependencies: finished with status 'done' #12 21.78 Getting requirements to build wheel: started #12 22.08 Getting requirements to build wheel: finished with status 'done' #12 22.09 Preparing wheel metadata: started #12 22.42 Preparing wheel metadata: finished with status 'done' #12 22.62 Collecting wheel #12 22.62 Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB) #12 22.65 Collecting jupyter #12 22.69 Downloading jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB) #12 23.32 Collecting jupyterlab #12 23.33 Using cached jupyterlab-3.1.6-py3-none-any.whl (8.6 MB) #12 23.44 Collecting nbresuse #12 23.45 Downloading nbresuse-0.4.0-py2.py3-none-any.whl (13 kB) #12 23.55 Collecting jupyterlab-topbar #12 23.62 Downloading jupyterlab_topbar-0.6.1-py3-none-any.whl (56 kB) #12 23.75 Collecting jupyterlab-system-monitor #12 23.77 Downloading jupyterlab_system_monitor-0.8.0-py3-none-any.whl (63 kB) #12 23.85 Collecting jupytext #12 23.86 Downloading jupytext-1.11.4-py3-none-any.whl (292 kB) #12 23.94 Collecting RISE #12 23.96 Downloading rise-5.7.1-py2.py3-none-any.whl (4.3 MB) #12 24.17 Collecting voila #12 24.19 Downloading voila-0.2.10-py3-none-any.whl (1.6 MB) #12 24.26 Collecting jupyterlab-sos #12 24.32 Downloading jupyterlab_sos-0.8.1-py3-none-any.whl (359 kB) #12 24.38 Collecting aquirdturtle_collapsible_headings #12 24.43 Downloading aquirdturtle_collapsible_headings-3.1.0-py3-none-any.whl (42 kB) #12 24.49 Collecting jupyterlab-execute-time #12 24.55 Downloading jupyterlab_execute_time-2.0.5-py3-none-any.whl (134 kB) #12 24.60 Collecting jupyterlab-skip-traceback #12 24.66 Downloading jupyterlab_skip_traceback-4.0.0-py3-none-any.whl (30 kB) #12 24.73 Collecting nbgitpuller #12 24.74 Downloading nbgitpuller-0.10.1-py2.py3-none-any.whl (15 kB) #12 24.83 Collecting jupyterhub #12 24.85 Downloading jupyterhub-1.4.2-py3-none-any.whl (3.6 MB) #12 24.94 Requirement already satisfied: packaging in /usr/lib/python3.9/site-packages (from jupyterlab-git==0.32.0) (20.9) #12 24.98 Collecting pexpect #12 24.99 Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB) #12 25.13 Collecting jupyter-server #12 25.13 Using cached jupyter_server-1.10.2-py3-none-any.whl (392 kB) #12 25.18 Collecting nbdime~=3.1 #12 25.21 Downloading nbdime-3.1.0-py2.py3-none-any.whl (5.1 MB) #12 25.35 Collecting nbformat #12 25.35 Using cached nbformat-5.1.3-py3-none-any.whl (178 kB) #12 25.46 Collecting pygments #12 25.47 Using cached Pygments-2.10.0-py3-none-any.whl (1.0 MB) #12 25.52 Collecting jupyter-server-mathjax>=0.2.2 #12 25.54 Downloading jupyter_server_mathjax-0.2.3-py3-none-any.whl (3.1 MB) ```

generates the following error

maxima-jupyter installation log

``` #18 [13/39] RUN git clone robert-dodier/maxima-jupyter.git /home/jovyan/maxima-jupyter #18 0.711 Cloning into '/home/jovyan/maxima-jupyter'... #18 DONE 1.4s #19 [14/39] WORKDIR /home/jovyan/maxima-jupyter #19 DONE 0.0s #20 [15/39] RUN export PYTHON_SITE=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])') && mkdir -p ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && cp maxima.js ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && patch ${PYTHON_SITE}/notebook/static/components/codemirror/mode/meta.js codemirror-mode-meta-patch && cp maxima_lexer.py ${PYTHON_SITE}/pygments/lexers/ && patch ${PYTHON_SITE}/pygments/lexers/_mapping.py pygments-mapping-patch #20 0.134 patching file /usr/lib/python3.9/site-packages/notebook/static/components/codemirror/mode/meta.js #20 0.137 patching file /usr/lib/python3.9/site-packages/pygments/lexers/_mapping.py #20 0.137 Hunk #1 FAILED at 217. #20 0.138 1 out of 1 hunk FAILED -- saving rejects to file /usr/lib/python3.9/site-packages/pygments/lexers/_mapping.py.rej #20 ERROR: process "/bin/sh -c export PYTHON_SITE=$(python -c 'import sysconfig; print(sysconfig.get_paths()[\"purelib\"])') && \tmkdir -p ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \tcp maxima.js ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \tpatch ${PYTHON_SITE}/notebook/static/components/codemirror/mode/meta.js codemirror-mode-meta-patch && \tcp maxima_lexer.py ${PYTHON_SITE}/pygments/lexers/ && \tpatch ${PYTHON_SITE}/pygments/lexers/_mapping.py pygments-mapping-patch" did not complete successfully: exit code: 1 ------ > [15/39] RUN export PYTHON_SITE=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])') && mkdir -p ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && cp maxima.js ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && patch ${PYTHON_SITE}/notebook/static/components/codemirror/mode/meta.js codemirror-mode-meta-patch && cp maxima_lexer.py ${PYTHON_SITE}/pygments/lexers/ && patch ${PYTHON_SITE}/pygments/lexers/_mapping.py pygments-mapping-patch: #20 0.134 patching file /usr/lib/python3.9/site-packages/notebook/static/components/codemirror/mode/meta.js #20 0.137 patching file /usr/lib/python3.9/site-packages/pygments/lexers/_mapping.py #20 0.137 Hunk #1 FAILED at 217. #20 0.138 1 out of 1 hunk FAILED -- saving rejects to file /usr/lib/python3.9/site-packages/pygments/lexers/_mapping.py.rej ------ Dockerfile:72 -------------------- 71 | 72 | >>> RUN export PYTHON_SITE=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])') && \ 73 | >>> mkdir -p ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \ 74 | >>> cp maxima.js ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \ 75 | >>> patch ${PYTHON_SITE}/notebook/static/components/codemirror/mode/meta.js codemirror-mode-meta-patch && \ 76 | >>> cp maxima_lexer.py ${PYTHON_SITE}/pygments/lexers/ && \ 77 | >>> patch ${PYTHON_SITE}/pygments/lexers/_mapping.py pygments-mapping-patch 78 | RUN curl -kLO beta.quicklisp.org/quicklisp.lisp && \ -------------------- error: failed to solve: process "/bin/sh -c export PYTHON_SITE=$(python -c 'import sysconfig; print(sysconfig.get_paths()[\"purelib\"])') && \tmkdir -p ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \tcp maxima.js ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \tpatch ${PYTHON_SITE}/notebook/static/components/codemirror/mode/meta.js codemirror-mode-meta-patch && \tcp maxima_lexer.py ${PYTHON_SITE}/pygments/lexers/ && \tpatch ${PYTHON_SITE}/pygments/lexers/_mapping.py pygments-mapping-patch" did not complete successfully: exit code: 1 Error: buildx failed with: error: failed to solve: process "/bin/sh -c export PYTHON_SITE=$(python -c 'import sysconfig; print(sysconfig.get_paths()[\"purelib\"])') && \tmkdir -p ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \tcp maxima.js ${PYTHON_SITE}/notebook/static/components/codemirror/mode/maxima/ && \tpatch ${PYTHON_SITE}/notebook/static/components/codemirror/mode/meta.js codemirror-mode-meta-patch && \tcp maxima_lexer.py ${PYTHON_SITE}/pygments/lexers/ && \tpatch ${PYTHON_SITE}/pygments/lexers/_mapping.py pygments-mapping-patch" did not complete successfully: exit code: 1 ```

noted in https://github.com/cameronraysmith/notebooks/issues/2

[robert-dodier]

Hmm, I think I see what the problem is, but I'm not sure how to handle it.

It looks like the problem is that the stuff in maxima-jupyter for code highlighting is based on a previous version of pygments, and pygments has been modified since maxima-jupyter stuff was put together, so they're inconsistent now.

On the face of it, it could be fixed by updating maxima-jupyter's idea of what the pygments mapping file looks like. But that would make maxima-jupyter incompatible with older versions -- maybe that's an important problem or maybe it's negligible; I don't know at this point.

Maybe there's a more lenient way to patch the mapping file. After all, all that's needed is to put maxima on the list.

Finally, another option is to ignore the error an keep going -- it's not a showstopper if the code highlighting doesn't work.

@yitzchak do you have any opinion on this issue?

[yitzchak]

There may be some way to add the lexer to pygments without modifying the system-level configuration. Would have to read the documentation though.

The other alternative is to see if we can get the Maxima stuff merged into CodeMirror and Pygments.

[robert-dodier]

Yeah, getting the Maxima stuff merged into CodeMirror and Pygments makes a lot of sense. Let me look into that.

[yitzchak]

Sounds good. Let me know if you need help.

[cameronraysmith]

Thank you. I had previously updated line numbers for the CodeMirror patch in my fork https://github.com/cameronraysmith/maxima-jupyter/commit/3d47c755074e34be3ba3a44c9fc052a89b4ba4ca , but as you all indicated, since this requires ongoing maintenance, it is not a very good solution.

[robert-dodier]

@yitzchak I've forked pygments and copied the Maxima lexer. You can see the fork here: https://github.com/robert-dodier/pygments

I think it's almost ready for a PR. The one part which I haven't figured out is that the instructions (https://pygments.org/docs/lexerdevelopment/) say that one should execute

$ pytest tests/examplefiles/maxima --update-goldens

but I find that provokes an error message, "pytest: error: no such option: --update-goldens", Likewse pytest3 instead of pytest gets the same message.

I suspect that's because of my ancient system (Ubuntu 14.04). I can't figure out a version number for pytest or pytest3; pytest --version and pytest3 --version both fail with no such option, and dpkg, pip list, and pip3 list don't seem to know about pytest/pytest3.

Can I ask you to try the --update-goldens command?

[cameronraysmith]

@robert-dodier I am happy to run this for you.

╭─ /data-cameron/jovyan/projects/pygments master ───────────────────────────────────────────────────────────────── 12s 10:36:15 PM ─╮
╰─❯ pytest tests/examplefiles/maxima --update-goldens                                                                              ─╯
======================================================== test session starts =========================================================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /data-cameron/jovyan/projects/pygments, configfile: pytest.ini
plugins: anyio-3.3.0
collected 1 item                                                                                                                     

tests/examplefiles/maxima/foo.mac .                                                                                            [100%]

========================================================= 1 passed in 0.03s ==========================================================
╭─ /data-cameron/jovyan/projects/pygments master ?1 ────────────────────────────────────────────────────────────────── 10:36:34 PM ─╮
╰─❯ pip freeze | grep pytest                                                                                                       ─╯
pytest==6.2.4
╭─ /data-cameron/jovyan/projects/pygments master ?1 ────────────────────────────────────────────────────────────────── 10:36:49 PM ─╮
╰─❯ g config --get remote.origin.url                                                                                               ─╯
https://github.com/robert-dodier/pygments.git

It looks like it is currently passing. Let me know if you need it run again.

[robert-dodier]

Cameron, thanks a lot. The pygments developer instructions to which I linked suggest that's supposed to generate an output file. Can you post the output, if any?

[cameronraysmith]

Hmm if I set --log-file=pygments-pytest.log and cat pygments-pytest.log after run the file exists but is empty.

@robert-dodier there is an output file at ./tests/examplefiles/maxima/foo.mac.output

pytest pygments foo.mac.output

``` '/*' Comment.Multiline ' ' Comment.Multiline 'S' Comment.Multiline 'F' Comment.Multiline ' ' Comment.Multiline 'b' Comment.Multiline 'u' Comment.Multiline 'g' Comment.Multiline ' ' Comment.Multiline '#' Comment.Multiline '3' Comment.Multiline '8' Comment.Multiline '2' Comment.Multiline '6' Comment.Multiline ':' Comment.Multiline ' ' Comment.Multiline '"' Comment.Multiline 'l' Comment.Multiline 'i' Comment.Multiline 'm' Comment.Multiline 'i' Comment.Multiline 't' Comment.Multiline ' ' Comment.Multiline 'r' Comment.Multiline 'e' Comment.Multiline 't' Comment.Multiline 'u' Comment.Multiline 'r' Comment.Multiline 'n' Comment.Multiline 's' Comment.Multiline ' ' Comment.Multiline 't' Comment.Multiline 'e' Comment.Multiline 'm' Comment.Multiline 'p' Comment.Multiline ' ' Comment.Multiline 'v' Comment.Multiline 'a' Comment.Multiline 'r' Comment.Multiline 'i' Comment.Multiline 'a' Comment.Multiline 'b' Comment.Multiline 'l' Comment.Multiline 'e' Comment.Multiline ' ' Comment.Multiline 'e' Comment.Multiline 'x' Comment.Multiline 'p' Comment.Multiline 'r' Comment.Multiline 'e' Comment.Multiline 's' Comment.Multiline 's' Comment.Multiline 'i' Comment.Multiline 'o' Comment.Multiline 'n' Comment.Multiline '"' Comment.Multiline ' ' Comment.Multiline '*/' Comment.Multiline '\n' Text '\n' Text '(' Punctuation 'kill' Name.Function ' ' Text '(' Punctuation 'q' Name.Variable ',' Text ' ' Text 'a' Name.Variable ',' Text ' ' Text 'x' Name.Variable ')' Punctuation ',' Text '\n' Text ' ' Text 'ctxt' Name.Variable ':' Operator ' ' Text 'newcontext' Name.Function ' ' Text '(' Punctuation ')' Punctuation ',' Text '\n' Text ' ' Text 'assume' Name.Function ' ' Text '(' Punctuation 'q' Name.Variable ' ' Text '>' Operator ' ' Text '0' Literal.Number.Integer ')' Punctuation ',' Text '\n' Text ' ' Text 'limit' Name.Function '(' Punctuation 'x' Name.Variable '^' Operator 'q' Name.Variable '/' Operator '(' Punctuation 'a' Name.Variable '*' Operator 'x' Name.Variable '^' Operator 'q' Name.Variable '-' Operator ' ' Text '1' Literal.Number.Integer ')' Punctuation ',' Text 'x' Name.Variable ',' Text 'inf' Name.Constant ')' Punctuation ')' Punctuation ';' Text '\n' Text "'" Operator 'limit' Name.Function '(' Punctuation '1' Text '/' Operator '(' Punctuation 'a' Name.Variable '-' Operator ' ' Text '1' Text '/' Operator 'x' Name.Variable '^' Operator 'q' Name.Variable ')' Punctuation ',' Text 'x' Name.Variable ',' Text 'inf' Name.Constant ')' Punctuation ';' Text '\n' Text '\n' Text 'tlimit' Name.Function '(' Punctuation 'x' Name.Variable '^' Operator 'q' Name.Variable '/' Operator '(' Punctuation 'a' Name.Variable '*' Operator 'x' Name.Variable '^' Operator 'q' Name.Variable '-' Operator ' ' Text '1' Literal.Number.Integer ')' Punctuation ',' Text 'x' Name.Variable ',' Text 'inf' Name.Constant ')' Punctuation ';' Text '\n' Text "'" Operator 'limit' Name.Function '(' Punctuation '1' Text '/' Operator '(' Punctuation 'a' Name.Variable '-' Operator ' ' Text '1' Text '/' Operator 'x' Name.Variable '^' Operator 'q' Name.Variable ')' Punctuation ',' Text 'x' Name.Variable ',' Text 'inf' Name.Constant ')' Punctuation ';' Text '\n' Text '\n' Text '(' Punctuation 'assume' Name.Function ' ' Text '(' Punctuation 'a' Name.Variable ' ' Text '>' Operator ' ' Text '0' Literal.Number.Integer ')' Punctuation ',' Text '\n' Text ' ' Text 'declare' Name.Function ' ' Text '(' Punctuation 'q' Name.Variable ',' Text ' ' Text 'integer' Name.Variable ')' Punctuation ',' Text '\n' Text ' ' Text 'limit' Name.Function '(' Punctuation 'x' Name.Variable '^' Operator 'q' Name.Variable '/' Operator '(' Punctuation 'a' Name.Variable '*' Operator 'x' Name.Variable '^' Operator 'q' Name.Variable '-' Operator ' ' Text '1' Literal.Number.Integer ')' Punctuation ',' Text 'x' Name.Variable ',' Text 'inf' Name.Constant ')' Punctuation ')' Punctuation ';' Text '\n' Text '1' Text '/' Operator 'a' Name.Variable ';' Text '\n' Text '\n' Text '(' Punctuation 'remove' Name.Function ' ' Text '(' Punctuation 'q' Name.Variable ',' Text ' ' Text 'integer' Name.Variable ')' Punctuation ',' Text '\n' Text ' ' Text 'declare' Name.Function ' ' Text '(' Punctuation 'q' Name.Variable ',' Text ' ' Text 'noninteger' Name.Variable ')' Punctuation ',' Text '\n' Text ' ' Text 'limit' Name.Function '(' Punctuation 'x' Name.Variable '^' Operator 'q' Name.Variable '/' Operator '(' Punctuation 'a' Name.Variable '*' Operator 'x' Name.Variable '^' Operator 'q' Name.Variable '-' Operator ' ' Text '1' Literal.Number.Integer ')' Punctuation ',' Text 'x' Name.Variable ',' Text 'inf' Name.Constant ')' Punctuation ')' Punctuation ';' Text '\n' Text '1' Text '/' Operator 'a' Name.Variable ';' Text '\n' Text '\n' Text '(' Punctuation 'remove' Name.Function ' ' Text '(' Punctuation 'q' Name.Variable ',' Text ' ' Text 'noninteger' Name.Variable ')' Punctuation ',' Text '\n' Text ' ' Text 'killcontext' Name.Function ' ' Text '(' Punctuation 'ctxt' Name.Variable ')' Punctuation ')' Punctuation ';' Text '\n' Text 'done' Name.Variable ';' Text '\n' Text ```

[robert-dodier]

Thanks, Cameron, I've committed the output as tests/examplefiles/maxima/foo.mac-output.

I've opened a PR to merge the Maxima lexer into pygments. See: https://github.com/pygments/pygments/pull/1885

I notice that in foo.mac-output, some items are labeled Text which I would not expect. In particular some instances of 1 are Text instead of Literal.Number.Integer, and , and ; are Text instead of Punctuation. I'm not going to worry about that, it works well enough as it stands, and if we can get the PR merged and installation working, I'll be pretty happy. We can circle back later to for fine tuning if there is interest.

[cameronraysmith]

Many, many thanks for your work here Robert!

[robert-dodier]

Just an update, the original PR for the pygments stuff wasn't accepted, but I've reworked it and requested another review. Maybe now it ill be okay.

[robert-dodier]

The PR to create a Maxima lexer for Pygments (https://github.com/pygments/pygments/pull/1885) was accepted, so I'll close this issue, although it will be some time (a few months, I was told) before the Maxima lexer is released as part of the next Pygments version, which will be 2.11.0 from what I gather.

I guess at some point, maybe now, maybe later, it makes sense to cut the Pygments stuff out of Maxima-Jupyter. Does anyone have a sense of when that should happen, now vs when the new Pygments is released?

[yitzchak]

Maybe we should update the patch file once more so the Docker build works.

[robert-dodier]

Maybe we should update the patch file once more so the Docker build works.

Okay by me, if someone can supply a patch that would be great. I don't think I can do it.

[yitzchak]

I'll see what I can do.