Lack of ipython cell magic code extractors

[cccs-jc]

Notebooks in vscode lack code extractor support for ipython magics. Here's an example with an %%sql magic.

1) The first issue is that the language/mime type of the code cell is not inferred by the use of %%sql magic. A user has to manually toggle the language of the code cell.

2) Once the code cell language is toggle extensions which handle the language are activated correctly. Syntax highlighting works and an SQL formatting extension makes its formatting menu available. However the text presented to the SQL formatter is the entire content of the cell and thus the extension formats the entire cell including the %%sql magic and its arguments.

This is also an issue for LSP servers. For example the sql-language-server will receive the text below which breaks proper parsing of the SQL statement.

%%sql postgresql://user:pass@localhost/db

SELECT     'hello'

In JupyterLab ipython magic are better handled. The jupyterlab-lsp project uses transclusions which are regex code extractors. These extractors determine the mime type of the cell and pull out the relevant code based on the cell magic detected.

https://github.com/jupyter-lsp/jupyterlab-lsp/blob/master/packages/jupyterlab-lsp/src/transclusions/ipython/extractors.ts

Extensions like LSP servers and formatting tools are presented text that is relevant to them.

SELECT     'hello'

This would enable this https://github.com/microsoft/vscode-jupyter/issues/4554#issuecomment-771187715

Is this something that is being considered?

[greazer]

This could be handled by each individual language service. But it's standard for jupyter notebooks to support jupyter specific magics beginning with %% on the first line only.

[cccs-jc]

Yes it can be handled by the individual language service however it requires each extension to make special cases for cell magics.

For example if you install an extension which will format SQL statements in .sql files they will not work in a jupyter notebook code cell with SQL language.

Because of this limitation I had to write my own SQL formatter extension which is aware of magics.

[greazer]

Thanks for making us aware of this problem @cccs-jc, but after a good amount of discussion, we've decided that doing anything more than encouraging language services and code-formatters to understand that they may have scenarios related to notebook use tends to resolve to hacky and/or fragile solutions. So for now we're not planning on making any further changes in the jupyter extension itself.

[cccs-jc]

Thanks for your consideration. I understand your point of view. I've written my own extension to get proper formatting in notebooks. I suppose over time relevant extensions will become notebook aware. cheers

[ahmed2m]

Is there any solution for the automatic cell language mode? When I open up a notebook full of sql "magic" cell (%%sql) I get tons of errors from pylance and I have to MANUALLY change each cell language to sql to avoid pylance linting

[cccs-jc]

as @greazer mention is it up to each extension to implement automatically "switching" cell mime type based on magic.

We built and use this project and get that automagic stuff https://github.com/CybercentreCanada/jupyterlab-sql-editor

let me know if that's of interest