search

quarto-dev / quarto-cli

Open-source scientific and technical publishing system built on Pandoc.
https://quarto.org
Other
3.88k stars 318 forks source link

`jupyter`: warn when kernel apparently doesn't support inline expressions (was: Stata inline code error) #10979

Open hugetim opened 2 weeks ago

hugetim commented 2 weeks ago

Bug description

My attempt to use Stata inline code results in an apparent Python error.

Steps to reproduce

Sample doc:

---
title: "Stata inline code"
format: html
jupyter: nbstata
---

```{stata}
disp 3

Testing inline {stata} disp 3.

I attempt to create a preview from within VSCode and get the following terminal output:
```bash
Starting nbstata kernel...Done

Executing 'stata-inline.quarto_ipynb'
  Cell 1/1: ''...Done
ERROR: 

'NoneType' object has no attribute 'get'

Expected behavior

No response

Actual behavior

No response

Your environment

Quarto check output

Quarto 1.5.57
[>] Checking versions of quarto binary dependencies...
      Pandoc version 3.2.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
      Typst version 0.11.0: OK
[>] Checking versions of quarto dependencies......OK
[>] Checking Quarto installation......OK
      Version: 1.5.57
      Path: C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\bin
      CodePage: 1252

[>] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[>] Checking LaTeX....................OK
      Tex:  (not detected)

[>] Checking basic markdown render....OK

[>] Checking Python 3 installation....OK
      Version: 3.10.12 (Conda)
      Path: C:/Users/tjhuegerich/AppData/Local/anaconda3/envs/nbstata_demo/python.exe
      Jupyter: 5.3.1
      Kernels: nbstata, python3

[>] Checking Jupyter engine render....OK

[>] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/
cscheid commented 1 week ago

Because this is a proprietary kernel, we can't really help much with its debugging. But you could use the advice here.

hugetim commented 1 week ago

Thank you. I hadn't seen that section of the docs. Unfortunately, the debug mode didn't add anything helpful---the closest thing is an additional line that says "- Error response received".

What might be helpful for me is the full traceback for the apparent Python error 'NoneType' object has no attribute 'get'

(I actually maintain the open source nbstata kernel. It calls the proprietary 'pystata' under the hood, but you can see how it does that here: https://github.com/hugetim/nbstata/blob/master/nbstata/kernel.py. edit: Oh, but now I see the problem is that you can't reproduce the issue. I'll see if I can reproduce with an "echo" kernel.)

Here's the temporary notebook file Quarto creates:

```bash { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "title: \"Stata inline code\"\n", "format: html\n", "jupyter: nbstata\n", "execute:\n", " debug: true\n", "---" ], "id": "8c10f86e" }, { "cell_type": "code", "metadata": {}, "source": [ "disp 3" ], "id": "455eb13a", "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Testing inline `{stata} disp 3`.\n" ], "id": "ba4bbe67" } ], "metadata": { "kernelspec": { "name": "nbstata", "language": "stata", "display_name": "Stata (nbstata)", "path": "C:\\Users\\tjhuegerich\\AppData\\Roaming\\jupyter\\kernels\\nbstata" } }, "nbformat": 4, "nbformat_minor": 5 } ```

And here's Quarto's jupyter log output:

```bash 2024-10-07 15:57:34,221 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:273 - starting notebook server subprocess 2024-10-07 15:57:36,578 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:278 - running notebook server subprocess 2024-10-07 15:57:36,579 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:106 - creating notebook server (tcp: C:\Users\tjhuegerich\AppData\Local\quarto\jt\468ba56d00cf61952166) 2024-10-07 15:57:36,582 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:132 - notebook server bound to port 57715 2024-10-07 15:57:37,118 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:34 - handling server request 2024-10-07 15:57:37,130 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\jupyter.py:65 - executing notebook 2024-10-07 15:57:37,137 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:164 - inside notebook_execute 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:186 - notebook was read 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:434 - { "name": "nbstata", "language": "stata", "display_name": "Stata (nbstata)", "path": "C:\\Users\\tjhuegerich\\AppData\\Roaming\\jupyter\\kernels\\nbstata" } 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:445 - No stata directory found in C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\lang\stata 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:446 - Will look for explicit quarto setup cell information in kernelspec dir 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:453 - No quarto_setup_cell file found in C:\Users\tjhuegerich\AppData\Roaming\jupyter\kernels\nbstata 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:454 - C:\Users\tjhuegerich\AppData\Roaming\jupyter\kernels\nbstata\quarto_setup_cell 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:134 - not using cache 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:208 - Will attempt to create notebook 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:210 - type of notebook: 2024-10-07 15:57:37,165 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:381 - Creating NotebookClient 2024-10-07 15:57:43,348 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:222 - NotebookClient created 2024-10-07 15:57:43,348 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 0 2024-10-07 15:57:43,352 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:507 - cell_execute with eval=True 2024-10-07 15:57:43,352 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:293 - Executed cell 0 2024-10-07 15:57:43,355 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:309 - Handling quarto metadata 2024-10-07 15:57:43,356 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:310 - { "id": "d99f783b", "cell_type": "code", "metadata": {}, "execution_count": 1, "source": "", "outputs": [] } 2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 1 2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:293 - Executed cell 1 2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 2 2024-10-07 15:57:43,357 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:507 - cell_execute with eval=True 2024-10-07 15:57:45,431 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:293 - Executed cell 2 2024-10-07 15:57:45,431 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:328 - Done 2024-10-07 15:57:45,431 - root - Level 25 - C:\Users\tjhuegerich\AppData\Local\Programs\Quarto\share\jupyter\notebook.py:278 - Executing cell 3 ```

hugetim commented 1 week ago

After setting QUARTO_PRINT_STACK, I can see this traceback now:

'NoneType' object has no attribute 'get'
ERROR: Error
    at renderFiles (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:78081:29)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async render (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:82929:21)
    at async Command.actionHandler (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:83077:32)
    at async Command.execute (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:8017:13)
    at async Command.parseCommand (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:7907:20)
    at async quarto (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118224:9)
    at async file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118244:9
    at async mainRunner (file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118128:9)
    at async file:///C:/Users/tjhuegerich/AppData/Local/Programs/Quarto/bin/quarto.js:118235:5
cscheid commented 1 week ago

(I actually maintain the open source nbstata kernel. It calls the proprietary 'pystata' under the hood, but you can see how it does that here: https://github.com/hugetim/nbstata/blob/master/nbstata/kernel.py. edit: Oh, but now I see the problem is that you can't reproduce the issue. I'll see if I can reproduce with an "echo" kernel.)

Hey, nice to meet you 👋 ! I'm sorry Quarto isn't working well in this scenario.

The code we use to execute inline expressions in a kernel is here:

https://github.com/quarto-dev/quarto-cli/blob/04524462d5653b57969db2daa49a0b4dfc13bc19/src/resources/jupyter/notebook.py#L544-L602

Concretely,

From there, you should be able to create a minimal example using only nbclient in your own kernel to test things more directly. Let me know if you need further info.

cscheid commented 1 week ago

@hugetim If you want to debug this further, I'd also highly recommend you install a development build of Quarto, which will give you better stack traces because we don't run a javascript bundle in the dev builds.

cscheid commented 1 week ago

If it helps further, it seems that user_expressions is implemented in the ipykernel as such:

https://github.com/ipython/ipykernel/blob/8d60e674db5c10738b65ced587aa76242f693dfd/ipykernel/ipkernel.py#L472-L476

hugetim commented 1 week ago

If it helps further, it seems that user_expressions is implemented in the ipykernel as such:

https://github.com/ipython/ipykernel/blob/8d60e674db5c10738b65ced587aa76242f693dfd/ipykernel/ipkernel.py#L472-L476

That is extremely helpful. I see I need to add support for user_expressions to the kernel, and that should be straightforward.

My guess is that nbstata's lack of support for user_expressions results in the error occurring here, trying to call 'get' on result, which will be None: https://github.com/quarto-dev/quarto-cli/blob/174d6ff48d368ad85cf9ec6c0a031f3c2406bae1/src/resources/jupyter/notebook.py#L581-L583 It may be helpful for others to check whether 'result' is None, raising an informative error if not.

Might I also suggest adding something about inline expressions' use of user_expressions to the docs here and/or adding basic "echo" user_expressions handling to the model Quarto kernel here?

cscheid commented 1 week ago

Might I also suggest adding something about inline expressions' use of user_expressions to the docs here and/or adding basic "echo" user_expressions handling to the model Quarto kernel here?

We definitely need that. Can you create a documentation issue here and a bug report on quarto_echo_kernel about this? Thanks.

cscheid commented 1 week ago

I've retitled this issue to track your other improvement suggestion, which is for us to warn when kernels do not appear to support user_expressions.

hugetim commented 1 week ago

... I see I need to add support for user_expressions to the kernel, and that should be straightforward. ...

With regard to the original (now superseded) issue here, I can confirm that after having added support for user_expressions to the nbstata Jupyter kernel, Stata inline code now works as expected.

cscheid commented 6 days ago

@hugetim Excellent. Thanks for letting us know!