Interpolation in footnotes

[ReubenJ]

Use case: I'd like to combine footnotes from this package with ShortCodes.jl's DOI to do something like:

md"""
As discussed in [^article], we want to do something.
"""

and further down the document:

md"""
[^article]: $(DOI(...))
"""

Resulting in something like

As discussed in [1], we want to do something.

[1]

Buettner, Florian; Diamanti, Evangelia; Fisher, Jasmin; Göttgens, Berthold; Haghverdi, Laleh; Jawaid, Wajid; Kouskoff, Valerie; Lilly, Andrew J; Macaulay, Iain; Moignard, Victoria; Nishikawa, Shin-Ichi; Piterman, Nir; Tanaka, Yosuke; Theis, Fabian; Wilkinson, Adam C; Woodhouse, Steven Decoding The Regulatory Network Of Early Blood Development From Single-Cell Gene Expression Measurements, Nature Biotechnology (2015) 10/f64rrc, cited by 337

It currently doesn't interpolate the DOI object (or any expression within the text) and looks something like this:

As discussed in [1], we want to do something.

[1] :(DOI("10.1038/nbt.3154"))

I guess it has something to do with the way the inner text is just taken verbatim here, and something would need to change there.

[eford]

That seems like a nice feature suggestion. Unfortunately, I'm not competent at JavaScript. I'd suggest that we ask some of the folks who contributed to https://github.com/JuliaPluto/PlutoUI.jl/issues/44 if they might be interested in helping here. @shashi, @dralletje, @mossr, @fonsp

On Mon, Sep 23, 2024 at 8:40 AM Reuben Gardos Reid @.***> wrote:

Use case: I'd like to combine footnotes from this package with ShortCodes.jl https://github.com/hellemo/ShortCodes.jl/tree/main's DOI to do something like:

md"""As discussed in [^article], we want to do something."""

and further down the document:

md"""[^article]: $DOI(...)"""

Resulting in something like

As discussed in [1], we want to do something.

[1]

Buettner, Florian; Diamanti, Evangelia; Fisher, Jasmin; Göttgens, Berthold; Haghverdi, Laleh; Jawaid, Wajid; Kouskoff, Valerie; Lilly, Andrew J; Macaulay, Iain; Moignard, Victoria; Nishikawa, Shin-Ichi; Piterman, Nir; Tanaka, Yosuke; Theis, Fabian; Wilkinson, Adam C; Woodhouse, Steven Decoding The Regulatory Network Of Early Blood Development From Single-Cell Gene Expression Measurements, Nature Biotechnology (2015) 10/f64rrc https://doi.org/10/f64rrc, cited by 337

It currently doesn't interpolate the DOI object (or any expression within the text) and looks something like this:

As discussed in [1], we want to do something.

[1] DOI("10.1038/nbt.3154")

I guess it has something to do with the way the inner text is just taken verbatim here https://github.com/JuliaPluto/PlutoTeachingTools.jl/blob/ca8cca84e7e0a96eee45ddd193f1b93072c29429/src/footnotes.jl#L17C11-L17C34, and something would need to change there.

— Reply to this email directly, view it on GitHub https://github.com/JuliaPluto/PlutoTeachingTools.jl/issues/57, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKDFQRCU52VT37GDSNLIELZYAD2RAVCNFSM6AAAAABOV7T23OVHI2DSMVQWIX3LMV43ASLTON2WKOZSGU2DENJQHA4TSNA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Eric Ford

[mossr]

Just as trying to interpolate variables in any other Markdown block (regardless of the footnote or not), you'll have to do the following:

Markdown.parse("""
[^article]: $(DOI(...))
""")

[ReubenJ]

Hmm, I don't think that's true. If you do the following:

md"""
$(1+1)
"""

It interpolates it correctly and it displays 2.

If I use Markdown.parse, it does interpolate, but I get the repr form of the DOI, not the show(MIME("text/html"), ...) form:

Markdown.parse("""
[^article]: $(DOI("10.1038/nbt.3154"))
""")

⬇️

[1] ShortCodes.DOI{String}("10.1038/nbt.3154")

I guess I could capture the output of show and then interpolate that into the footnote, but a cleaner solution would be nice.

[dralletje]

Or, our recommendation, use @markdown from MarkdownLiteral.jl

@markdown """
[^article]: $(DOI("10.1038/nbt.3154"))
"""

[eford]

I got UndefVarError:@markdownnot defined when I tried that. But

MarkdownLiteral.@markdown """
[^article]: $(DOI("10.1038/nbt.3154"))
"""

did work for me. Or import MarkdownLiteral: @markdown.

[dralletje]

Yes that is correct

[eford]

@ReubenJ if that works for you, then would you like to add this to the example notebooks, so others can easily see how to use this pattern?

[ReubenJ]

Sure, I've updated the example to include this (see my fork, can open a PR if this is useful for others), but I don't understand why that's necessary when the interpolation seems to work just fine with $(1+1), for example. Any help, @dralletje?

Also, it doesn't matter much, but the end of the generated citation gets an extra line break before the DOI when using @markdown. I'm not sure why that's the case...

[eford]

Thanks. I created a PR. Maybe the two Markdown parsers treat the new line output by DOI differently?

[dralletje]

I'm not sure why we are wondering about the difference in Markdown parsers, because for me the Markdown.parse(...) method does not work:

@markdown is made to work with HTML interpolations (the ideal way of showing things in Pluto), that is why it works here (was surprised to see that DOI does not have a string representation at all)

---

Also, it doesn't matter much, but the end of the generated citation gets an extra line break before the DOI when using @markdown. I'm not sure why that's the case...

Not sure if you are using a different Pluto setup, but for me all [^link] thingies get a newline:

Even more so, because DOI creates an html <div> which also creates a newline, even if other text would show inline.

---

Sure, I've updated the example to include this (see my fork, can open a PR if this is useful for others), but I don't understand why that's necessary when the interpolation seems to work just fine with $(1+1), for example. Any help, @dralletje?

I'm unsure if something changed, or I'm using a wrong Julia version or anything, but md"[^article]: $(1 + 1) doesn't even create what I expected:

For some reason, the built-in markdown parser errors when doing [^article]: ..... This should be fixable, but the reason it works with MarkdownLiteral is because we use "more robust" interpolation.

You might notice how @markdown has nice interpolation syntax highlighting! That's because in Julia string macros like md"..." does not support interpolations. md"..." cheats this by doing their own version of interpolations and that causes some problems. @markdown does actual Julia string interpolation, so has "more power" to fit the interpolations in correctly.

[eford]

I'm not sure why we are wondering about the difference in Markdown parsers, because for me the Markdown.parse(...) method does not work:

Sorry, I was replying to @ReubenJ's observation above that there was an extra new line in the footnote when using MarkdownLiteral.

[eford]

Even more so, because DOI creates an html <div> which also creates a newline, even if other text would show inline.

I think that's the answer. Personally, I prefer it shown inline. I'd suggest @ReubenJ create an issue for ShortCodes.jl to resolve this.

[dralletje]

Even more so, because DOI creates an html <div> which also creates a newline, even if other text would show inline.

I think that's the answer. Personally, I prefer it shown inline. I'd suggest @ReubenJ create an issue for ShortCodes.jl to resolve this.

It isn't, normal text is also shown on a separate line. This is just how Pluto shows footnotes 🤷🏽 You can style it if you really want, but ShortCodes switching from a <div> to a <span> won't change a thing.

[ReubenJ]

@eford and I are referring to this newline:

[dralletje]

Ahaaaa, figured I was missing something!

So yeah, this IS because of the markdown parser, the way @markdown works, and something specific about the DOI html. (Been on a 50 minute deep dive 😂)

Julia native md"..." does not allow html inside [^article]: ... text at all. @markdown does, because of CommonMark, BUT there is something very specific going on: The DOI html has newlines, which makes it that the result we put into the markdown parser is

[^article]: <div>Buettner, Florian; Diamanti, Evangelia; Fisher, Jasmin; Göttgens, Berthold; Haghverdi, Laleh; Jawaid, Wajid; Kouskoff, Valerie; Lilly, Andrew J; Macaulay, Iain; Moignard, Victoria; Nishikawa, Shin-Ichi; Piterman, Nir; Tanaka, Yosuke; Theis, Fabian; Wilkinson, Adam C; Woodhouse, Steven <em>Decoding The Regulatory Network Of Early Blood Development From Single-Cell Gene Expression Measurements</em>, Nature Biotechnology (2015)
<a href=https://doi.org/10/f64rrc>10/f64rrc</a>, cited by 337</div>

And Commonmark panics! So it puts an extra <p> around <a href=https://doi.org/10/f64rrc>10/f64rrc</a>, cited by 337</div>... and that creates the extra spacing!!

Then, Pluto also does some odd styling itself around the footnotes...

So where to create issues to fix this? I think the problem isn't with DOI. they are returning fine HTML. CommonMark is not really to blame, html like this isn't supposed to parse like anything useful anyway (no other markdown parser I could find did anything CLOSE to this useful)

So I think this falls on MarkdownLiteral.jl for how it handles html inlining, but not sure if we can fix this without breaking other uses.. Though I'm willing to give it a try

---

For now you can fix it with a little wrapper:

# ╔═╡ 84d36bcf-cbf2-4ea7-9a94-b29f0dae512f
struct DOI2
    cite
end

# ╔═╡ 0570067f-6376-443f-a595-29d5f0909d92
function Base.show(io::IO, ::MIME"text/html", x::DOI2)
    text= repr("text/html", DOI(x.cite))
    write(io, replace(text, "\n" => " ", "<div>" => "<p>", "</div>" => "</p>"))
end

It's not pretty (in code)... but it works

(We need the "<div>" => "<p>" because of some pluto style quirks)

[eford]

Thanks for diving in! If you think there's a chance MarkdownLiteral.jl might update to make this go away, I'm inclined to wait a while and see.

If it becomes clear that's not a viable option, then I think we could have PlutoTeachingTools.jl import ShortCodes.jl internally, export its own DOI using the trick you've demonstrated, and reexport the other short codes as is. Not ideal, but less painful than asking every instructor to copy Base.show code into their notebooks.

[fonsp]

Hey all! So far we looked into fixing the problem that this syntax does not work:

md"""
[^article]: $(DOI("asdfasdf"))
"""

But for citations my dream would be to have functionality designed specifically for citation management, similar to how bibtex works.

So you could do:

md"""
This is great![^knuth]
"""

And to show the list of used references, you just call a function that returns a pretty display of citations.

list_references(bibtex=read("my_library.bibtex", String))

or

list_references(
    "knuth" => "https://doi.org/10.1038/nbt.3154",
    "pluto" => "https://doi.org/10.5281/zenodo.4792401",
)