What options exist for extending markdown cells to support reports on multiple kernels? [Was: Can customized kernel hijack markdown cells?]

[BoPeng]

I am working on a kernel where the underlying workflow system can generate reports based on results of other steps. For example,

! ## samples
! There are ${nsamples} samples with names ${sample_names}

would generate

## samples
There are 2 samples with names sampleA, sampleB

I would like to store these report lines in jupyter as markdown cells but currently the kernel can only handle script cells through do_execute. Is there anyway that I can hijack markdown cell, change its content (expand ${samples} etc) before it is rendered?

[Carreau]

I would like to store these report lines in jupyter as markdown cells but currently the kernel can only handle script cells through do_execute. Is there anyway that I can hijack markdown cell, change its content (expand ${samples} etc) before it is rendered?

No, not currently. We had the discussion a long time ago, and want something similar and generic across languages. The current conclusion is that we will not allow it before markdown/common mark have a syntax that allows extensions, otherwise you end up writing your own parser, what we don't have the bandwith to do.

You can (of course) decide to monkey patch the notebook with the custom extension, we can't prevent you to do that.

[BoPeng]

I went through the discussions and I have to agree that jupyter should really try to resolve this issue to make the reports more dynamic, and thus more useful. I disagree with your point that jupyter has to provide a language-neutral solution because variable and expressions from different languages take different forms and it is impossible to settle down with a common syntax. All jupyter needs to do is provide a do_markdown api to give each kernel a chance to handle markdown cells differently. In my case, I would simply do

def do_markdown(self, cell):
    cell.source = interpolate(cell.source, self.my_dict)
    return super(SoS_Kernel, self).do_markdown(cell)

In the case of python, with the introduction of new format string to version 3.6, there is no reason to think of any other syntax.

[takluyver]

Hiding code cells is also a possibility, which would make it easier to do such things as the output of a hidden code cell.

[jasongrout]

There is a hack to do this: If you use a PlaceProxy widget, you can replace any html id tag with a widget. So you could in your markdown cell do:

## samples
There are <span style="display:inline-block;" id="nsamples"></span> samples with names <span style="display:inline-block;" id="sample_names"></span>

and then in your code cell, do

from ipywidgets import PlaceProxy, Label
from IPython.display import display

nsamples = Label('5')
sample_names = Label("'name1'")
display(PlaceProxy(nsamples, selector="#nsamples"))
display(PlaceProxy(sample_names, selector="#sample_names"))

Then if you set, for example, nsamples.value='10', that should update the markdown.

This is really fragile - it depends on the markdown being rendered to html on the page when the widget is displayed. It doesn't persist to the notebook file. But it is a kludgy way to do what you're asking.

[jasongrout]

@BoPeng - markdown cells are never sent to the kernel in the current notebook. They are rendered completely in the browser.

[BoPeng]

I understand it now, but this does not have to mean it cannot be changed. I share the opinion with many others (https://github.com/ipython/ipython/pull/2592) that RStudio is miles ahead of Jupyter for generating pretty reports and this is one of the key features that are lacking. The issue was discussed back in 2012 and I do not understand why it was never taken seriously (apologize if I am wrong here).

[BoPeng]

@takluyver Yes, that would more or less work, but it is not ready, right? (I can see https://github.com/jupyter/notebook/issues/534). I do not know how that operates, but users can enter

! ## samples
! There are ${nsamples} samples with names ${sample_names}

as code, evaluate it, and hide the cell. It is not as convenient as entering

## samples
There are ${nsamples} samples with names ${sample_names}

as markdown though.

[takluyver]

It's not that we didn't take it seriously, it's that it's not obvious how to make it work intuitively, and there are thousands of things we want to do.

Apart from the syntax issue Matthias mentioned (we really don't want to go making more ad-hoc special symbols for Markdown), there's an interface question - do we expand the variables in a Markdown cell when we render it, or keep them up to date as code is executed? The former is technically much simpler, but Markdown cells, unlike code cells, have no visible code or prompt number to indicate that they might need to be re-executed. Should they disappear or stay when the kernel is restarted, the page is reloaded, or the notebook is converted to another format? A format like knitr has this much easier, because it has a clear source document and output document. If that's what you want, have a look at knitpy, or the cross-language tool dexy. Doing this interactively is more challenging)

(I'm sure you can come up with answers for the questions above, but my point is that the answers aren't obvious - something has to be hashed out between different use cases and priorities)

[BoPeng]

we really don't want to go making more ad-hoc special symbols for Markdown

I read arguments from both sides but I am on the side that we are NOT creating another special symbol for markdown. Because each language has its own syntax and there is no way we can find a commonly accepted one, the decision on a syntax should be made at the kernel level. The kernels decide what special symbols/syntax to use and how to translate them to proper markdown syntax . I mean, Jupyter does not have to add or understand anything. It simply feed the markdown cell to the kernel (if do_markdown is defined) and render the returned markdown text.

there's an interface question - do we expand the variables in a Markdown cell when we render it, or keep them up to date as code is executed? The former is technically much simpler, but Markdown cells, unlike code cells, have no visible code or prompt number to indicate that they might need to be re-executed.

No. The markdown cells are evaluated with shift-enter and users can re-evaluated them if needed. The users could run-all-cells to generate a final updated document.

Should they disappear or stay when the kernel is restarted, the page is reloaded, or the notebook is converted to another format?

The markdown cells would be hide-source cells with both versions saved (like code cells), but only the evaluated version is displayed. This is exactly what users are seeing now: the markdown code is hidden unless we enter the cell.

A format like knitr has this much easier, because it has a clear source document and output document. If that's what you want, have a look at knitpy, or the cross-language tool dexy. Doing this interactively is more challenging)

I failed to understand why it is so challenging if we accept a bit of inconvenience (e.g. manual re-evaluation of markdown cells) caused by the interactive nature of workflow.

[BoPeng]

BTW, I understand @Carreau's argument that we are adding special syntax if we introduce special syntax to markdown cells of the python kernel, which can in theory cause existing documents fail to render. However, with the introduction of new hook (do_markdown), it costs almost nothing to introduce derived kernels such as

class MdPythonKernel(PythonKernel):
    def do_markdown(self, cell):
          return process(cell.source)

and let users decide which kernel they want to use. Again, it does not have to be jupyter's job to decide on special syntax for any kernel.

[jasongrout]

@BoPeng - what about doing this as a cell magic?

%%interactivemd

## samples
There are ${nsamples} samples with names ${sample_names}

Cell magics are sent to the backend where the markdown can be rendered to html and sent back to the browser. Then the only piece missing is easily hiding the input and easily showing the input if you want to edit the source.

[BoPeng]

Yes, this would certainly work. I just need to decide if I should add another way to specify report because users can already enter ! lines in code cells, and %report just adds another special code cell.

[Carreau]

Because each language has its own syntax and there is no way we can find a commonly accepted one,

That untrue, we want to insert value in markdown, so it's enough to find a caracter which work in markdown.

the decision on a syntax should be made at the kernel level

That an opinion, not a truth, and no after long reflexion we think that doing that at kernel level is a bad idea for many reason:

• it's blocking,
• it would needs the code to be reavaluated by nbconvert
• it requires each kernel to implement a markdown parser and renderer,
• Initial prototypes of the notebook were rendering markup cell server side, it was a nightmare.

I mean, Jupyter does not have to add or understand anything. It simply feed the markdown cell to the kernel (if do_markdown is defined) and render the returned markdown text.

That would require an enormous change in protocol, and break the main abstraction that kernels are only for computation, and there cannot be "if do_markdown" is defined, as this is a protocol you should conform to, and multiple client can be connected. Plus as far as we know there are alternative frontends that do not use markdown at the markup language. You cannot for that on them.

No. The markdown cells are evaluated with shift-enter and users can re-evaluated them if needed. The users could run-all-cells to generate a final updated document.

Again that's your opinion, some people do not like this solution, plus this would have some consequence in the front-end like blocking the rendering of MD cells, which is basically a no-no.

I failed to understand why it is so challenging if we accept a bit of inconvenience (e.g. manual re-evaluation of markdown cells) caused by the interactive nature of workflow.

The concept is interesting, and does make sens in some use case, and it is perfectly reasonable in some workflow. We did went through some of your ideas a few years ago, and after reflexions and prototypes we really things these are not the solution we like. Plus we have to maintain backward compatibility.

So I think you underestimate the cost, advantages/drawbacks of such solutions, there are likely some pros and cons, and good ideas. Like I personally, of opinion that Shift-Entrer requerying variable to the kernel for a markdown cell is a good idea, with another key to render without changing the current value is one thing I like. But I recognize that for user it is highly confusing, and for non modal frontend, it would be a nightmare to come up with the right semantics.

So we would be happy to see prototype, but it's likely not a direction we will take, at least not soon.

Would be happy to proven wrong if you want to hack a prototype.

[jasongrout]

Yes, this would certainly work. I just need to decide if I should add another way to specify report because users can already enter ! lines in code cells, and %report just adds another special code cell.

I'm not sure what you mean here. What do ! lines have to do with an %%interactivemd cell, and what is %report?

[BoPeng]

I'm not sure what you mean here. What do ! lines have to do with an %%interactivemd cell, and what is %report?

Sorry for the confusion. In SoS, reports are marked by lines with !, which can be mixed with other statements. I had hoped to separate them as markdown cells but now I need to process them in code cells, in which they already work in the ! form. Using %%interactivemd (or another name %report that I would prefer) just assumes all content in the code cell should be prefixed by !.

As a side question, there is no difference between line and cell magics in jupyter notebook, right? Because notebook would not evaluate a line starts with %magic directly with enter, I just used % for all sos magics.

[BoPeng]

Because each language has its own syntax and there is no way we can find a commonly accepted one,

That untrue, we want to insert value in markdown, so it's enough to find a caracter which work in markdown.

I think we have some serious misunderstanding on how dynamic markdown works so let me explain. When Jupyter sees

## samples
There are ${nsamples} samples with names ${sample_names}

Your idea is to identify ${nsamples} and ${sample_names} as dynamic, send nsamples and sample_names to the kernel, get their string representations, and replace ${nsamples} and ${sample_names} with their values.

But this would not work if we are to allow arbitrary expressions because Jupyter does not know the syntax of the underlying language and does not know where to stop the pattern. For example, if a language happens to use ${ } for its string (just for fun),

There are ${ ${this is my string} }

Jupyter would send an invalid expression ${this is my string to the kernel. This is why I said that it is necessary for the kernel to parse the cell text with its own syntax. Of course you can disallow expressions (note that Rmarkdown allows it) but if jupyter needs to interact with the kernel for variables, I see no reason not to send the whole cell.

[Carreau]

The point is not to send arbitrary expression. You can't send arbitrary expression that have side effects, that would bread the assumption that rendering a text has no side effects.

[takluyver]

I've already mentioned that I don't want to make ad-hoc extensions to Markdown syntax, so the idea of encouraging kernels to each make their own ad-hoc extensions, and then implement mini parsers for them, is definitely not something I want.

However, why not try doing this as a Javascript extension? You could hook the rendering of a Markdown cell and send it to your kernel to do what you want with.

[BoPeng]

The point is not to send arbitrary expression. You can't send arbitrary expression that have side effects, that would bread the assumption that rendering a text has no side effects.

I do not get it. As soon as you send something to the kernel, there can be side effects. It does not matter what Jupyter think it is, variable or expression, the kernel can see it differently. For example, if you send ${ls} to bash, it might as well consider it a command and execute it.

So the resistance is on the dynamic markdown itself, not what syntax to use and what send to the kernel, and I do not understand why. It is true that rending a text should have no side effect, but we are rending dynamic text with side effect.

[BoPeng]

I've already mentioned that I don't want to make ad-hoc extensions to Markdown syntax, so the idea of encouraging kernels to each make their own ad-hoc extensions, and then implement mini parsers for them, is definitely not something I want.

I personally see nothing wrong for a kernel to make use of such features to generate better notebooks and I regret to see that your decision is based on your personal preference, not users' need. To me, not able to use `r 1+1 ` equivalence to RStudio was one of the reasons why I gave up ipython/notebook a few years ago and I am surprised to see this has not been resolved after so many years.

However, why not try doing this as a Javascript extension? You could hook the rendering of a Markdown cell and send it to your kernel to do what you want with.

I have close-to-zero knowledge on javascript... and I need to make sure that that is a proper way to implement such a kernel feature when I develop my own kernel.

[Carreau]

I do not get it. As soon as you send something to the kernel, there can be side effects. It does not matter what Jupyter think it is, variable or expression, the kernel can see it differently

IIRC the protocol explicitly define user_variables and user_expressions, ls, alias and ls variable can perfectly be distinguished by the IPython kernel during the namespaces lookup.

Fair enough for bash it seem different but nothing prevent you from doing a bash kernel that check wether something has side effects.

So the resistance is on the dynamic markdown itself, not what syntax to use and what send to the kernel, and I do not understand why. It is true that rending a text should have no side effect, but we are rending dynamic text with side effect.

No the resistance is on: 1) variable having side effects. If we implement that, we want the assumption that variable will not have side effects. Kernels can have side effects if they want to, but we will make the assumption they dont. 2) And on syntax, maintaining a syntax on top of markdown is a lot of work. And wether it's kernel that render, or frontend, we get the bugs report, and handle user requests.

I personally see nothing wrong for a kernel to make use of such features to generate better notebooks and I regret to see that your decision is based on your personal preference, not users' need. To me, not able to use r 1+1 equivalence to RStudio was one of the reasons why I gave up ipython/notebook a few years ago and I am surprised to see this has not been resolved after so many years.

In the end, we spend time implementing and maintaining, while user need is relevant, it is weighted by usage, and the burden on our time, difficulty and maintenance, and other deadline we have. We are happy to revisit with more manpower as this is something we want to do.

Right now despite the fact that we seem like a big project, we are a small team with many things to work on. If someone actually regularly involved in the project that contribute show an interest in pushing that forward, we'll likely revisit the decision, but currently no member of the core team have the time, and many of our discussions make us to believe that there is more work involved than it appears at first look.

I have close-to-zero knowledge on javascript... and I need to make sure that that is a proper way to implement such a kernel feature when I develop my own kernel.

Happy to help you getting acquainted with it. None of us were javascript developers when we started this.

[jasongrout]

@BoPeng - you're wanting to generate reports, right? Can you use knitpy and write .pymd files? Is that equivalent to what rstudio offers?

[BoPeng]

@BoPeng - you're wanting to generate reports, right? Can you use knitpy and write .pymd files? Is that equivalent to what rstudio offers?

I am using multiple languages and would like to have an environment that I can work with R, python, shell etc, with ability to generate reports. The project we are working on is called script of scripts, which is a workflow system with a Jupyter kernel. The kernel can execute SoS steps, and can start and switch between sub-kernels (e.g. IRKernel) in the same notebook. You can have a preview of a jupyter notebook with SoS kernel at https://github.com/BoPeng/SOS/blob/master/examples/example.ipynb (no installation of sos is needed thanks to github).

[jasongrout]

Are you wanting to have text that displays in the notebook showing the current values of variables interjected into text (e.g., a cell that you evaluate and you get text/values intermingled)? Or are you wanting to have a one-step conversion from a notebook to a separate report (which actually may be a notebook, or could be a separate file) that has values substituted in various places?

Can you write a quick simple mockup of what you want the final result to be? Like what would it look like in the context of your example.ipynb?

[BoPeng]

For SoS, reports from each step will be evaluated (with variables and expressions replaced by their values) and collected to form a final report file. There is no requirement on the format of this report but it is usually in markdown format and users can either process it outside of SoS, or add a step to their workflow to call, for example RMarkdown to generate reports in HTML or PDF formats.

I am having a hard time to decide how reports should behave in Jupyter. Ideally, it could be dynamic markdown cells that can be evaluated by SoS. In this way the Jupyter notebook will display results of SoS steps as a report with the ! lines hidden under the markdown, and there is no real need to post-process reports generated by SoS, because the notebook itself or its HTML output are good enough in most cases. This is what the example notebook is showing, without the dynamic part.

Now that Jupyter does not allow dynamic markdown, showing reports in markdown cells is not good enough so I will have to handle them in code cells (e.g. with a %report magic, or with ! lines). The report will be processed with HTML outputted as result. The notebook will look like

---- code cell ---
%report
## samples
There are ${nsamples} samples with names ${sample_names}
---- result ---
samples (in header format)

There are 2 samples with names a.txt, b.txt
------------------

As you can see, although Jupyter can still be used to preview the report, the notebook does not look like a report anymore and it is likely that users have to post-process reports generated by SoS, which is something I would really like to avoid.

[takluyver]

your decision is based on your personal preference, not users' need

When I say that "I don't want" ad hoc extensions to Markdown, I'm using it as a shorthand for "I don't believe this is a good thing for users in the long term". It means the markdown in notebooks would be less generalisable, relying on a specially extended engine to render it correctly, and it's a source of potential future bugs as we have to maintain extra parsing machinery.

You seem to be pushing hard for a solution that will work for your particular use case, without seriously working out how it fits into the notebook model as a whole. Your "bit of inconvenience caused by the interactive nature of workflow" is indicative. The notebook is an interactive tool, and it's not trying to be knitr. Doing something like this well requires thinking about interactivity, and that's a harder problem than batch processing.

We are happy to help you do what you want as an extension, but I don't see this as something that should go into the protocol and the notebook application as it stands.

[BoPeng]

You seem to be pushing hard for a solution that will work for your particular use case, without seriously working out how it fits into the notebook model as a whole.

As I have said, I gave up ipython notebook a few years ago because it lacks some features as a notebook tool. I am certainly not alone as many users in the 2012 thread shared the same concern.

Although I have learned a lot about Jupyter in the past few days through the development of the SoS kernel, I am not familiar with Jupyer core and I am certainly not in a position to 'seriously working out how it fits into the notebook model'. However, from a user's point of view, I am working in an interactive environment with python or R, or some other language, if a code cell can be evaluated, if the markdown cell also need be evaluated (Ctrl-Enter), it is only nature for a markdown cell to integrate better with the shell and present what is happening there. This is my perception of the "notebook model", although it does not agree completely with the model of Jupyter.

[takluyver]

My concern is that there's no clear indication on a Markdown cell of when it was executed or even that it might need to be reexecuted at all.

Also, I think markdown cells that have to be 'run' to be up to date could soon be implemented with a cell magic that produces Markdown output and hides the code cell. This won't require any changes to the protocol or the notebook (beyond the cell hiding that we're already planning).

I think a neater approach for the notebook might be to use HTML tags like <span data-expression="x"></span>, and then keep track of which variables are displayed and update them as code is executed (there's already a user_expressions field in execute messages to get back representations). These could then be stored in metadata, or in the companion file we're also planning. This is more complex, but I think it could provide a better user experience.

[willingc]

All, I've updated the title of this issue because the original title implies a security issue on quick scan of titles (i.e. "hijack"). I've updated the title of this issue to better reflect the discussion within the issue.

[BoPeng]

My concern is that there's no clear indication on a Markdown cell of when it was executed or even that it might need to be reexecuted at all.

There is no need for guessing with the Ctrl-Enter re-evaluation. I actually prefer manual re-evaluation because I dislike evaluation behind my back.

Also, I think markdown cells that have to be 'run' to be up to date could soon be implemented with a cell magic that produces Markdown output and hides the code cell. This won't require any changes to the protocol or the notebook (beyond the cell hiding that we're already planning).

Does this work for all kernels? What I learned so far is that kernels have to implement their own magics.

I think a neater approach for the notebook might be to use HTML tags like , and then keep track of which variables are displayed and update them as code is executed (there's already a user_expressions field in execute messages to get back representations). These could then be stored in metadata, or in the companion file we're also planning. This is more complex, but I think it could provide a better user experience.

There is a lot of work to enter <span ...>. Also, there is no way for Jupyter to know if the variable has been updated unless the kernel also keeps such a list and notify Jupyter after the variable changes. I therefore think it is simpler to have manual re-evaluation.

[BoPeng]

If someone actually regularly involved in the project that contribute show an interest in pushing that forward, we'll likely revisit the decision, but currently no member of the core team have the time, and many of our discussions make us to believe that there is more work involved than it appears at first look.

I guess this represents the decision from the team. About 15 years ago when I was a graduate student with plenty of spare time, I contributed a lot to an open source project for similar reasons, but right now I do not have the bandwidth to contribute to Jupyter. Hopefully someone else with more time and skill can contribute such a feature.

I think I will continue to use markdown cells for static contents such as headers for SoS notebooks, and use code cells for dynamic reports. It is not pretty (e.g. cell 4 and 5 of the updated sos notebook example but the coming hide-cell feature can hopefully be the remedy.

Finally, I appreciate your hard work for making such a wonderful platform and your patience to explain to me the difficulties. I apologize if I sounded a bit harsh in my responses, but there was certain no hard feelings there.

[Carreau]

There is a lot of work to enter <span ...>. Also, there is no way for Jupyter to know if the variable has been updated unless the kernel also keeps such a list and notify Jupyter after the variable changes. I therefore think it is simpler to have manual re-evaluation.

That's why one of my prototype was doing, but with a {{}} syntax in markdown to avoid the work to write <span> and it was storing repr of variable in metadata, so that you could either: 1) render the MD with placeholders if kernel busy. 2) update the variable in place without re rendering the full markdown once they values are availables.

I never decided whether to update explicitly, or automatically, and technically you could have 2 syntax that each would have had a different behavior. like The current value of x is {{x}} and will update, and for a first step we can take x=[|x|] but change later. ( I just don't want to write a MD parser that add that)

Finally, I appreciate your hard work for making such a wonderful platform and your patience to explain to me the difficulties. I apologize if I sounded a bit harsh in my responses, but there was certain no hard feelings there.

No worry, it is always hard to convey the tone in writing, and we are really happy to see you creating the SOS kernel, with Jupyter getting traction we can hopefully have some people interested in tackling this problem and each conversation like these is alway a good source of information that help us to shape what it needed and hopefully get a solution at some point.

[parente]

At the spring dev meeting, there was a brief discussion about making calls for proposals (CFP) to request help from people with time and expertise in investigating and implementing big ticket features (e.g., internationalization). I get a sense this idea falls into the same basket.

I don't think spending time writing up big, formal CFPs is the right approach at the moment. But is there something we can be doing to make these research topics / big ideas / help wanted more discoverable by people who are looking to dive in to something big in Jupyter?

Off the cuff: apply a tag like "research project", aggregate that tag on a nice ideas.jupyter.org page by calling the github API. (Of course building the latter requires work too, so socializing a GitHub query to find all issues with the tag might be a simple place to start.)

[willingc]

@parente Nice idea. I've gone ahead and tagged this issue with your suggested terms.

[abalter]

Simple solution 1) Project variables into the javascript namespace using AJAX. Obviously, this would be troublesome for something like a dataframe, or even a large list or such. So, this is not really a great solution--it would become idiosyncratic and inconsistent.

Simple solution 2) Generate the dynamic text in python, but then have the ability to have it rendered as markdown. Now this seems possible. Maybe there could be a new kind of cell called a "Template" cell in which anything that goes to stdout would get rendered as markdown.

In non-template cell,

    print("""
## This is  some text
1. Item 1
1. Item 2
"""

Would get rendered literally.

## This is  some text
1. Item 1
1. Item 2

In a Template cell it would get rendered as

This is some text

1. Item 1
2. Item 2

[BoPeng]

A major concern here is that there is no consensus on the sigil to use for variable interpolation, and even if there is a consensus (e.g. use ${ }), there is no way to expose variables to the frontend without help from the kernel. I believe that it should be the kernel's decision on how to evaluate the expressions and interpolate the text so Jupyter only need to send the text to the kernel. In this way, a bash kernel will interpolate ${ }, and a R kernel would interpolate something similar to Rstudio.

[abalter]

Oh, and I didn't really make clear how my idea addresses using dynamic formatting based on variables. Here is how it would work:

Code

a = 10
print("# Something about a")
print("The value of `a` is **{}**".format(a))

Non-Template Cell Output:

# Something about a The value of `a` is **10**

Template Cell Output:

Something about a

The value of a is 10

[BoPeng]

I see your point. This is indeed a brilliant method to avoid the sigil choice problem. Users are free to produce any text in any language-specific way and there is no need for any specific kernel support.

If I understand correctly, all that is needed is that

1. The cell would be sent to the kernel and be evaluated like any other statement.
2. The output will be rendered as MD.
3. The input would be hidden like a markdown cell and be opened with double-click.

It is not as easy as string interpolation though. I mean, instead of writing

The result is `r res`.

users would have to write

paste('The result is ', res)

to display

The result is 10

[abalter]

Right, exactly. Just as in a templating language for websites. If in a template cell, only stdout is printed, and it is rendered as markdown. Up to the user to make sure they send what they actually want to stdout.

[BoPeng]

Given that it is unrealistic to have all kernels to support a new string interpolation feature, your proposal sound like a nice compromise. Are you going to implement it or start a new ticket to propose to the core developers? Rstudio has had inline string interpolation for many years, and Rstudio notebook already has nice multi-lanaguge support (something my SoS Jupyter kernel is working on), I believe Jupyter should move fast on features like this to catch up.

[abalter]

Oh golly, I'm not nearly a capable enough programmer to tackle it. And I don't know any of Jupyter's innards. I think I need to lame out and start a feature request ticket. But if someone could line me out on part of the task, I could give it a try. And yes, Jupyter needs to keep up.

[Carreau]

Hi all,

The discussion about that kind of things started about 4 years ago (OMG, it feels like I started contributed like yesterday), when things were still IPython notebook, and had a prototype. ( https://github.com/ipython/ipython/pull/2592)

While we would really like to have such solution, the implication are really touching everything, and so far I believe that we won't commit to accepting something like that unless we are sure there is a long term engagement.

We do miss though an essay on pro, cons, gotchas, advantage and state of the art about what is currently existing. Having something like that would already be a really good first step.

Thanks ! -- M

On Mon, Mar 6, 2017 at 8:13 AM, Ariel Balter notifications@github.com wrote:

Oh golly, I'm not nearly a capable enough programmer to tackle it. And I don't know any of Jupyter's innards. I think I need to lame out and start a feature request ticket. But if someone could line me out on part of the task, I could give it a try. And yes, Jupyter needs to keep up.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/help/issues/41#issuecomment-284444589, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUez_qBN51vCfWs2sQPPKc31Qhtb2eGks5rjDCqgaJpZM4IeO6x .

[BoPeng]

Could you elaborate on what @abalter's proposal would touch? It is to my understanding that this proposal is the least intrusive one throughout the discussions because no kernel support is required. This kind of cells behaves on the frontend as markdown cells (execute to render MD, hide input, double-click to show code) and the only difference is that the cells are sent to the kernel and are evaluate as regular cells.

Actually SoS has tried to implement some sort of active markdown (https://github.com/vatlab/SOS/issues/390), but failed because of my lack of knowledge on Jupyter frontend and JS skills. Perhaps this can be a new direction that we can try.

[abalter]

Here is an idea: What if it was simply a type of magic:

%%template
do
template
stuff

[Carreau]

Could you elaborate on what @abalter's proposal would touch? It is to my understanding that this proposal is the least intrusive one throughout the discussions because no kernel support is required. This kind of cells behaves on the frontend as markdown cells (execute to render MD, hide input, double-click to show code) and the only difference is that the cells are sent to the kernel and are evaluate as regular cells.

Yes.

Among other, it would change the notebook spec, which we are not against but is a lot of work. You need to update all the frontends to support that.

Moreover you need the kernel to produce the cell, which is problematic for things like nbconvert. Early prototype of the notebook were using Rest With server side rendering and it ended up being close to intractable.

You will of course need to have the kernels adopt your formatting, and be able to render whatever correctly. We already have issues with user typing non-standard markdown and using esoteric extensions, if kernel authors have the slightest mistake in their rendering code, then notebooks will become "wrong", and it's going to be a mess. Think also that we can't guaranty kernel-evaluation will not have side effects.

There is not much difference either from use a Code Cell that ends with display(Markdown()) , or even %%markdown at least in using IPython, and hiding the codecell at conversion time. some people even have other custom magics like %%jinja, or %%jade. That's already possible now, you just need to tweek a bit your nbconvert config/css.

So while things like that seem "Obvious" and "simple" we had long discussions, prototypes, they have large consequences on maintenance. And while this is something we want, we'd like to do it right, and that's means not only thinking about it, but also following up on it, maintaining it, discussing with publisher what they would accept, what they need.

[BoPeng]

In the end I implemented @abalter's idea in the SoS kernel with a render magic. Basically, the %render magic captures the standard output (or return value) of a cell (which in the case of SoS can be any subkernel), and process it with Markdown, HTML etc from module IPython.display.

This does not work at all like a Markdown cell. It is more troublesome to use but it is more powerful (e.g. support HTML, SVG, and other formats) and flexible. The bottom line is that it allows the generation of markdown content in any kernel using any method, which of course includes string interpolation. With the help of Jupyter extensions such as hide-code, users can hide the input of the cell so only the generated output is displayed. I could add an option --hide-input to the %render magic but then I would have to handle nbconvert and unhide cell interface, so I guess it is easier to leave this task to other extensions.