Magic cell languages vs magic cell commands

[mwouts]

Following #513 and #622, I'd like to think a bit about how magic cells are/should be handled.

Here are my expectations

1. If a cell has a magic cell that is a language, say e.g. R, that differs from the main notebook language, say e.g. python, then
   • it is entirely commented out in the .py file
   • it appears as a ```R code cell in the .md file
2. Cells with a magic cell that is not a language should appear verbatim in the .md file. The cell magic should always be commented out in the .py file. For some cell magics like %%time or %%capture it may not be necessary to comment out the cell content in the .py file. For some other cell magics like %%script, commenting out the full cell is required.

I think (but would like to check) that we are actually doing that.

Currently the distinction between 1. and 2. is made based on jupytext.languages._JUPYTER_LANGUAGES (script is one of them), plus the configurable custom_cell_magics. Do we cover all the use cases?

[lucharo]

Hello, is there an expected date for the support of custom non python magic (e.g. %%sql)? I have seen in another issue that R cells from rpy2 are already supported.

[mwouts]

Hi @lc5415 , custom magics are supported since Jupytext 1.6.0. Sorry I see it's not yet documented... see this comment for how to use them.

[lucharo]

Thanks for the reply @mwouts , I will try this tomorrow and I guess it should work. Is there a known way to set this up globally? And is there a way to set it up programmatically? Perhaps depending on the current environment/kernel? Custom magics could be detected automatically by comparing the current magics vs the default ones? (This may be too much/too specific, I'm just wondering about the options here)

[mwouts]

Is there a known way to set this up globally?

Yes, the option can be set in a jupytext.toml file, as you will see in the link above.

The file can be local to your folder, or even global, cf. https://jupytext.readthedocs.io/en/latest/config.html#jupytext-configuration-file

[lucharo]

Not working with double magic

Hi @mwouts ,using custom_cell_magics works well, unfortunately it does not work with several magic operators in a single cell: in my use case I have a cell like

%%time
%%sql
SELECT *
FROM table1

I would like this to be be commented too as the cell contains a non pythonic cell magic and I think it is a sensible default behaviour, what do you think?

[mwouts]

Hi @lc5415 , thanks for reporting the double magic case! I agree with you, the cell should be fully commented out in this case. I'll try and see if I can fix this in the next release.

[fm75]

I was surprised when jupytext did not include the output from the %%timeit magic. See this for a more complete discussion of what I discovered today. My sense is that on conversion from ipynb to markdown I would want the output "stream" included.

Is this the correct place for this information, or should it be a separate issue?

[mwouts]

Hi @fm75 , thanks for asking.

As @chrisjsewell and @choldgraf mentioned in the other discussion, the outputs are never stored in the Jupytext files (and this is not related to magic or not magic commands).

You were probably expecting something closer to the output of jupyter nbconvert ? Indeed this is an area where jupytext is very different from nbconvert - Jupytext focuses on the part of the notebook that the user wrote, not the part that was generated by the machine and which is often thousand times larger. Of course I understand that not having outputs can be troublesome, so depending you what you are trying to achieve, my recommendations are

• use jupyter nbconvert if you are looking for a one way conversion .ipynb to .md
• use Jupytext's paired notebooks (synchronized .ipynb + .md files) if you want to edit either the .ipynb (with outputs) or the .md file