Missing output of code cells

[cionx]

Chapters 5.9 (Functors) and 6.6. (Randomized Testing with QCheck) are currently missing all output from code cells. No other chapter seems to have this problem.

---

I don’t know why this problem occurs, but the following may help:

To better understand the problem I tried building the book on my local machine with the provided make file. Strangely enough, I get four chapters without code cell output, including the two chapters mentioned above:

ERROR: Execution Failed: /home/cionx/cs3110-textbook/src/chapters/correctness/randomized.md
ERROR: Execution Failed: /home/cionx/cs3110-textbook/src/chapters/ds/promises.md
ERROR: Execution Failed: /home/cionx/cs3110-textbook/src/chapters/modules/functors.md
ERROR: Execution Failed: /home/cionx/cs3110-textbook/src/chapters/basics/functions.md
ERROR: Couldn't find cache key for notebook file src/chapters/basics/functions.md. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/cionx/cs3110-textbook/src/_build/html/reports/functions.log
ERROR: Couldn't find cache key for notebook file src/chapters/correctness/randomized.md. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/cionx/cs3110-textbook/src/_build/html/reports/randomized.log
ERROR: Couldn't find cache key for notebook file src/chapters/ds/promises.md. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/cionx/cs3110-textbook/src/_build/html/reports/promises.log
ERROR: Couldn't find cache key for notebook file src/chapters/modules/functors.md. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/cionx/cs3110-textbook/src/_build/html/reports/functors.log

The given log files don’t mention the actual problem, but at least state where it occurs:

functions.log

``` Traceback (most recent call last): File "/home/cionx/.local/lib/python3.10/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution executenb( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 1204, in execute return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 663, in async_execute await self.async_execute_cell( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 965, in async_execute_cell await self._check_raise_for_error(cell, cell_index, exec_reply) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 862, in _check_raise_for_error raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content) nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ let rec zfact_aux n acc = if Z.equal n Z.zero then acc else zfact_aux (Z.pred n) (Z.mul acc n);; let zfact_tr n = zfact_aux n Z.one;; zfact_tr (Z.of_int 50) ------------------ : ```

functors.log

``` Traceback (most recent call last): File "/home/cionx/.local/lib/python3.10/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution executenb( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 1204, in execute return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 663, in async_execute await self.async_execute_cell( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 965, in async_execute_cell await self._check_raise_for_error(cell, cell_index, exec_reply) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 862, in _check_raise_for_error raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content) nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ #use "topfind";; #require "ounit2";; open OUnit2;; ------------------ : ```

promises.log

``` Traceback (most recent call last): File "/home/cionx/.local/lib/python3.10/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution executenb( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 1204, in execute return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 663, in async_execute await self.async_execute_cell( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 965, in async_execute_cell await self._check_raise_for_error(cell, cell_index, exec_reply) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 862, in _check_raise_for_error raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content) nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ let p, r = Lwt.wait();; ------------------ : ```

randomized.log

``` Traceback (most recent call last): File "/home/cionx/.local/lib/python3.10/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution executenb( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 1204, in execute return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 663, in async_execute await self.async_execute_cell( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 965, in async_execute_cell await self._check_raise_for_error(cell, cell_index, exec_reply) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 862, in _check_raise_for_error raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content) nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ QCheck.make QCheck.Gen.int ------------------ : ```

By looking at the quoted code sections we can see that each one relies on #use "topfind" and #require:

functions.md

````markdown ```{code-cell} ocaml :tags: ["remove-cell"] #use "topfind";; ``` ```{code-cell} ocaml :tags: ["remove-output"] #require "zarith.top";; ``` ```{code-cell} ocaml let rec zfact_aux n acc = if Z.equal n Z.zero then acc else zfact_aux (Z.pred n) (Z.mul acc n);; let zfact_tr n = zfact_aux n Z.one;; zfact_tr (Z.of_int 50) ``` ````

functors.md

````markdown ```{code-cell} ocaml :tags: ["remove-cell"] #use "topfind";; #require "ounit2";; open OUnit2;; ``` ````

randomized.md

````markdown ```{code-cell} ocaml :tags: ["remove-cell"] #use "topfind";; ``` ```{code-cell} ocaml :tags: ["remove-output"] #require "qcheck";; ``` (… some lines without code blocks featuring QCheck …) ```{code-cell} ocaml :tags: ["hide-output"] QCheck.make QCheck.Gen.int ``` ````

promises.md

````markdown ```{code-cell} ocaml :tags: ["remove-cell"] #use "topfind";; ``` ```{code-cell} ocaml :tags: ["remove-output"] #require "lwt";; ``` ```{code-cell} ocaml let p, r = Lwt.wait();; ``` ````

Searching for all uses of #require throughout the book (with grep -r '#require' src/chapters/) reveals that these are practically the only code cells containing #require: the only other occurrence is in modules/toplevel.md. But there, none of the following code cells actually tries to access the required module (OUnit2). If we were to add such a code cell ourselves, then this would introduce an additional building error at precisely that location:

toplevel.md

Changing in the markdown code ````markdown ```{code-cell} ocaml :tags: ["remove-cell"] #use "topfind";; ``` ```{code-cell} ocaml :tags: ["remove-output"] #require "ounit2";; ``` Now you can successfully load your own module without getting an error. ```ocaml open OUnit2;; ``` ```` the last code block to a code cell, resulting in ````markdown ```{code-cell} ocaml :tags: ["remove-cell"] #use "topfind";; ``` ```{code-cell} ocaml :tags: ["remove-output"] #require "ounit2";; ``` Now you can successfully load your own module without getting an error. ```{code-cell} ocaml open OUnit2;; ``` ```` will introduce a new error during the build process: ``` ERROR: Execution Failed: /home/cionx/cs3110-textbook/src/chapters/modules/toplevel.md ERROR: Couldn't find cache key for notebook file src/chapters/modules/toplevel.md. Outputs will not be inserted. Last execution failed with traceback saved in /home/cionx/cs3110-textbook/src/_build/html/reports/toplevel.log ``` The log file point to exactly the part that we changed: ``` Traceback (most recent call last): File "/home/cionx/.local/lib/python3.10/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution executenb( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 1204, in execute return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 663, in async_execute await self.async_execute_cell( File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 965, in async_execute_cell await self._check_raise_for_error(cell, cell_index, exec_reply) File "/home/cionx/.local/lib/python3.10/site-packages/nbclient/client.py", line 862, in _check_raise_for_error raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content) nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ open OUnit2;; ------------------ : ```

Given these observations, I strongly suspect that the problems on my machine are caused by the use of these toplevel commands in code cells.

---

But I don’t know if the problems in the official online version have the same cause: it would seem strange to me if only two of the four files were affected.

I tried looking at the output of the last GitHub Action to get more information, but the build artifacts (which may or may not have contained the error log files) are not available anymore, and the still-available console output doesn’t seem to contain any build messages.

[clarksmr]

@cionx Thanks! Yes, the version of the book currently deployed is missing output in those two sections. But when I build it locally I get the output. So there are two questions:

1. Why doesn't it build right for you?
2. Why is the deployed version wrong?

The latter is probably related to some similar problem that my colleague @justhsu is experiencing but hasn't noticed yet. He's the one who deployed last.

For 1, can you start by confirming that you have the required packages installed in the OPAM switch you're using to build the textbook, and that the switch is based on OCaml 4.14.0, and that the kernelspec is correctly configured? I'm curious about this, because of the error you get in functions.log, which could suggest that zarith is not installed.

[clarksmr]

(meanwhile I have deployed from my own local build to temporarily fix the output in those two sections. FYI @justhsu)

[justhsu]

Thanks @clarksmr. I think I hit this problem because I wasn't in the right opam switch. I am not sure if that is the same issue that @cionx is hitting.

[cionx]

I had all the required packages (ounit2, qcheck, menhir, zarith and jupyter) installed, and could also successfully run the code in utop.

I deinstalled all required packages, removed the kernel (using jupyter kernelspec uninstall), and then reinstalled everything following the instructions from BUILDING.md. This seems to have fixed the problem, which I have been unable to reproduce since then.

I did, however, diverge from the instructions in BUILDING.md at two points:

• I didn’t have to modify display_name in the config file kernel.json from the default name ("OCaml textbook") to "OCaml". It only seems to matter that the kernel name specified in the kernelspec section (at the top of most markdown files) matches the kernel name as shown by jupyter kernelspec list.

• I couldn’t build the project within the given conda environment:

  Error logs

  ``` [...] building [mo]: targets for 0 po files that are out of date building [html]: targets for 96 source files that are out of date updating environment: executing outdated notebooks... failed Extension error (myst_nb.execution): Handler for event 'env-get-outdated' threw an exception (exception: (SQLAlchemy 2.0.4 (/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages), Requirement.parse('sqlalchemy~=1.3.12'))) Traceback (most recent call last): File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/sphinx/events.py", line 94, in emit results.append(listener.handler(self.app, *args)) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/myst_nb/execution.py", line 76, in update_execution_cache _stage_and_execute( File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/myst_nb/execution.py", line 279, in _stage_and_execute execute_staged_nb( File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/myst_nb/execution.py", line 311, in execute_staged_nb executor = load_executor("basic", cache_base, logger=LOGGER) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/jupyter_cache/executors/base.py", line 78, in load_executor execute_cls = entry_points[0].load() File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2476, in load self.require(*args, **kwargs) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2499, in require items = working_set.resolve(reqs, env, installer, extras=self.extras) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/pkg_resources/__init__.py", line 815, in resolve dist = self._resolve_dist( File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/pkg_resources/__init__.py", line 861, in _resolve_dist raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.VersionConflict: (SQLAlchemy 2.0.4 (/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages), Requirement.parse('sqlalchemy~=1.3.12')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/jupyter_book/sphinx.py", line 171, in build_sphinx app.build(force_all, filenames) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/sphinx/application.py", line 330, in build self.builder.build_update() File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 286, in build_update self.build(to_build, File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 300, in build updated_docnames = set(self.read()) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 371, in read for docs in self.events.emit('env-get-outdated', self.env, added, changed, removed): File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/sphinx/events.py", line 102, in emit raise ExtensionError(__("Handler %r for event %r threw an exception") % sphinx.errors.ExtensionError: Handler for event 'env-get-outdated' threw an exception (exception: (SQLAlchemy 2.0.4 (/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages), Requirement.parse('sqlalchemy~=1.3.12'))) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/cionx/.conda/envs/textbook/bin/jupyter-book", line 10, in sys.exit(main()) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/jupyter_book/cli/main.py", line 322, in build builder_specific_actions( File "/home/cionx/.conda/envs/textbook/lib/python3.9/site-packages/jupyter_book/cli/main.py", line 530, in builder_specific_actions raise RuntimeError(_message_box(msg, color="red", doprint=False)) from result RuntimeError: =============================================================================== There was an error in building your book. Look above for the cause. =============================================================================== make: *** [Makefile:17: html] Error 1 ```

  The problem seems to be that ~/.conda/envs/textbook/lib/python3.9/site-packages/jupyter_cache-0.4.2.dist-info/METADATA contains the line Requires-Dist: sqlalchemy (~=1.3.12), whereas the current version of SQLAlchemy, which is also the version installed in textbook, is 2.0.4.

  It could fix/circumvent this problem in three ways:

  • Manually installing a suitable version with conda install sqlalchemy~=1.3.12.
  • Adding the requirement sqlalchemy~=1.3.12 to environment.yml before creating the environment.

  Both of these approaches seem to downgrade from version 2.0.4 to 1.3.23 (even though I’d expect version 1.3.24).

  • Not using the textbook environment or even miniconda at all. (This relies on me having jupyter-book installed on my system.)

  I find it strange that this error even occurs, as I would expect conda to automatically install the required version when creating the environment.

---

I only opened this issue to report the missing output in the online version, so from my point of view, this issue can be closed now. (But I thought that I should still report that/how I fixed the problem on my machine, and that there is possibly a problem with the provided conda environment, at least on some machines.)

[clarksmr]

Thanks @cionx! I can confirm that if I recreate the conda environment on my machine, I experience the same error as you.

If I pin jupyter-book to 0.11.2 (rather than let it go up to 0.13.2, as it does when I recreate the environment today), then the book seems to build successfully -- without the sqlalchemy errors.

So I'm going to commit that pin, close this issue, and open a new issue to upgrade the jupyter-book version.

[clarksmr]

Pinned in fd885d1

[clarksmr]

@justhsu You'll want to rebuild your conda environment to pick up that pin before you deploy again.