Hugo shortcodes are not preserved

[jemus42]

Examples:

• https://github.com/jemus42/hugodown-testing/blob/master/public/hugodown-rmarkdown/index.html#L179
• https://github.com/jemus42/hugodown-testing/blob/master/public/hugodown-rmarkdown/index.html#L286

I assume the issue is that " and ' are replaced with " etc., where the page shows {{< pkg “hugodown” >}}.

I hope it's possible to work around this without having to resort to solutions like blogdown::shortcode(), which inserts preserving comments <!--html_preserve--> that do not work well with shortcodes that return inline-output (like a <span>), see the first example here https://github.com/jemus42/hugodown-testing/blob/master/public/hugodown-rmarkdown/index.html#L274-L279 which is displayed with additional linebreaks like here.

Blogdown's .Rmarkdown format renders markdown which preserves shortcodes, see for example here: https://hugodown-testing.netlify.app/blogdown-rmarkdown/#shortcodes (source HTML here, source .Rmarkdown here).

If the solution is a regex to detect/preserve shortcodes (I have not investigated what blogdown does there), please note that there are two kinds of shortcodes that may appear in the post source:

Shortcodes returning markdown: {{% mdshortcode %}} Shortcodes returning HTML: {{< htmlshortcode >}}

[hadley]

I'll take a quick look, but unfortunately I'm unlikely to be able to put much time into it, because I think you're much better off using inline R code than hugodown, since that reduces your coupling with hugo.

[apreshill]

FYI, we are heavily using shortcodes in the tidymodels.org site, as the Hugo version of pandoc div boxes (like with bookdown).

[jemus42]

Thanks for taking a look.

Regarding the coupling with hugo, we appear to have different goals in mind – I would very much like to go all in on hugo, as it would allow me the freedom to use all the neat hugo features in both .md and .Rmd posts, whereas inline R code of course would not be usable in .md.
But that would be a separate discussion, and I'm fine with hugodown having different priorities than me in this regard :)

[hadley]

@apreshill you can use pandoc div syntax for that now 😄

@jemus42 I am all in on R 😄 I'll certainly take a look, but by and large the focus of hugodown is to insulate you from hugo as much as possible (partly so that you can easily repurpose blog posts into other Rmd formats).

[jemus42]

@hadley I see! Well I'd like you to know I feel somewhat misguided by hugodown being named after hugo and the "focussed purely on Hugo websites" bit in the README 😅

I kid of course, but the repurposibility of blog posts across alternative site generators is what blogdown is focused on (e.g. Jekyll, Hexo), so I thought this is where hugodown differs --- but that's good to clarify!
I'll be less picky about what I'd like hugodown to be, currently it just feels odd that both blogdown and hugodown represent differently imperfect solutions for me personally :)

[hadley]

It seems like there are at least three types of shortcode:

• {{% code params %}}
• {{% code %}}markdown text{{% /code %}}
• {{ <code params >}} A bunch of code here {{< /code >}}

But the documentation is pretty hard for me to parse. Protecting hugodown shortcodes from pandoc manipulation is going to be challenging, so if I do provide support its likely to be similar to what blogdown does, although I'd probably wrap it up a little differently so that each short code gets its own corresponding R function.

@jemus42 can you explain bit more why you prefer {{< pkg "hugodown" >}} to `r pkg("hugodown")`. They seem about equally (un)pleasant to type, and using R means that you can take advantage of your existing function writing skills. I see you mentioned being able to write .md above, but what's the disadvantage of using .Rmd with no code chunks?

@apreshill are there other shortcodes that you use? I'm pretty sure the pandoc div syntax (which I have already confirmed works) will be just as pleasant an editing experience.

[apreshill]

I do use other shortcodes. I often use the built-in ones for embedding tweets, gists, and YouTube videos. The academic theme I use for my personal site ships with additional ones, especially embedding Google slides (used here for teaching).

From the perspective of Hugo users and developers, shortcodes are part of the theme, and one of the few ways a theme developer has to allow Hugo code inside the content/ files themselves (i.e., below the YAML instead of in the YAML). They are also a nice way for community members to contribute, by adding shortcodes and exchanging on the discourse forums. Most are transferable across themes.

[jemus42]

Regarding the shortcode syntax, maybe I can help clarify:

There are two types of shortcodes regarding their delimiters, either {{< shortcode >}} for html output or {{% shortcode %}} for markdown output. Then there's shortcodes that enclose other content, those have an opening and an ending shortcode, where the ending shortcode contains a /, e.g. {{% /shortcode %}} or {{< /shortcode >}}.
My assumption was that it was possible to protect everything that's between {{% and %}} or {{< and >}} respectively, which is why I neglected to mention the case of {{% /shortcode %}}.

Regarding {{< pkg "hugodown" >}} vs. `r pkg("hugodown")`: Both are annoying to type, I grant you that. I'm German, and `'s are terrible due to dead keys and having to double-tap the key just to get one, and áccidental àccents… I digress. Typing comfort is not a concern in my case because I like using tools like Alfred for snippet management, so can quickly insert even complex shortcodes without having to think about it --- but I digress again 😅.

My mindset is "Hugo first", because using .md as a default instead of .Rmd with no code chunks means I benefit from

1. Hugo's blazing fast site generation (this is a huge point -- I love both knitr and pandoc with all my heart, but it's slooow compared to hugo, why waste that speed?)
2. Not having to worry about re-knitting older posts (in case I tweaked the theoretical pkg() function, for example). hugo always updates everything, fast.
3. Not having to worry about compatibility of whatever system renders my .Rmd with the system that builds the actual HTML and rest of the site, i.e. Hugo (Note that in the past I used Jekyll + manual knitting, blogdown with .Rmd and now blogdown with .Rmarkdown). There's alway's some trade-off, and some weird hack I need. MathJax or KaTeX may be good examples there.
4. Using other people's shortcodes and sharing my own, which might benefit not only R/blogdown/hugodown users, but Hugo-users in general 4.2 Using .Rmd with no code at all would generate a (mostly?) identical .md, which just feels wrong to me as it would be a needless duplication in case of a purely text-based post with maybe a {{< figure >}}, but I concede that's only worth 20% of an actual point 😁
5. If everything was .Rmd, I'd have to remember to re-render before pushing to GitHub to let netlify/my server rebuild my site after I just fixed the 15th consecutive typo in that blog post I published yesterday and am routinely finding typos in because I'm bad at words (okay, this one might be personal 😅) -- point being: No .Rmd, no R dependency (for that particular post)! (which is of course irrelevant for sites like tidyverse.org I imagine, where every post has R code)

In my "perfect world", hugodown would work similiar to blogdown's .Rmarkdown in that its output is fairly customizable (through knitr hooks) and entirely hugo-friendly.

Shortcodes work without extra steps, and I can use customizations like knitr hooks to have R chunks output plots into hugo's figure shortcode, which means plots are structurally the same as regularly included figures not generated by R.
I can then, for example, tweak the figure shortcode to make use of native lazy-loading by just adding one attribute.
I can also use a source hook to have R code chunks wrapped inside Hugo's highlight shortcode, which means full(er) control of the HTML structure and chroma-features like line highlighting and line numbers (as seen in my previous issue). The latter of course is not possible when embracing downlit (which I'm torn on, as you might imagine -- I love auto-linking functions etc., but I also really like shortcodes and consistency between R chunks and non-R chunks)

Making use of hugo shortcodes also means I can easily externalize shortcodes into a separate repository and include it as a hugo module / theme component across different sites, where other sites don't even have to be blogdown/hugodown sites (I talked about shortcode externalization here). Sure, you could put your "R-shortcodes" in a package, but then you're somewhat doomed to rewrite/duplicate many nifty hugo shortcodes in R, and of course a plain Hugo site wouldn't benefit from that.

Not to mention that many themes ship with shortcodes that provide theme-specific benefits, like this theme which provides shortcodes for visually distinct "highlight" or "note" boxes (something I have also adapted for my blog). Sure, you could re-create them in plain R and generate the same HTML from R, but that seems like unnecessary overhead.

Oh, I almost forgot: Shortcodes have access to the full range of Hugo's templating magic, so you can use shortcodes to reference and work with all kinds of site content and pages for a huge amount of flexibility if harnessed.

Once again, I'm sure it would technically be possible to do that with R, but at some point I'd have to ask:
Why use / focus on Hugo if you're not making use of some of its coolest features?

…and while I've been writing this way too long testament to my love of shortcodes, I see that Alison also has made a very good point with a much better efficiency, so I think I'll have to stop here - sorry for the long post 😔 (I'm just confusingly enthusiastic about all this).

[hadley]

Unfortunately I don't have the time to respond to your full ode to the shortcode, but I did want to respond to your numbered bullet points because that's easy:

1. In my experience with tidyverse.org with .Rmd posts that have little R code, the updates already feel very fast — you press cmd + shift + K and moments later the view updates.

2. With hugodown, you don't need to re-knit older posts, unless you want to re-run the R code.

3. I think unfortunately that this is the nature of the game; when writing code that goes into a page with existing css/js/etc, you have to design around it. I think you allude to this later with theme specific shortcodes.

4. R packages 😄

5. This doesn't seem to affect me too much because I have just trained myself to press cmd + shift + K instead of Cmd + S.

But I appreciate you taking the time to explain your viewpoint, and I think I now understand what you and @apreshill like about shortcodes, and while I think it would be great if hugodown could support them, I think it's going to be technically very challenging — either I am going spend far more time than I think it's worth, or I will have build a fragile solution that is going to break in edge cases. (I can revisit this decision if pandoc eventually adds a macro/extensibility system.)

hugodown is fundamentally an R package, and I believe that relying on existing mechanisms for code reuse in the R ecosystem is better long term approach. I think most R users who want to make a blog will not want to invest heavily in learning the details of hugo, and would instead be better served by relying on skills (like inline R code) that they can leverage in other situations. This means I've convinced myself that hugodown doesn't need to do more to support hugo shortcodes, and I'll not be pursing a general solution.

@apreshill for the specific short codes you need in tidymodels, we can talk about how to wrap them up in documented R functions.