Splice a code snippet from a text/code file

[jebej]

It might be useful to have the option to include a .jl file in a page and display the code. This would make it easy to have examples in an example folder, but still be able to display them in the documentation. There might already be a way to do that in html however, but I do not know how.

This is just a nice to have, as it is possible to do this simply by copying the code over and keeping it synced manually.

[mortenpi]

Nope, not currently possible, but would be something that could be done if someone needs it. Sounds like a new ```@-block (@include?). But I would leave the implementation as up for grabs though.

[s-celles]

Hello,

I was also looking for a similar feature to include sample code to documentation (but keeping it as a .jl file) I asked recently on Gitter without any positive result. You might have a look at this Discourse discussion https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304

Kind regards

[KristofferC]

The way you do it right now is:

````@eval
Markdown.parse("""

$(readstring("PATH_TO_FILE"))

""")

[s-celles]

Does it enable syntax color highlighting for Julia ?

[tpapp]

I am trying to implement this, but need a bit of help understanding the code.

1. Is it correct that the only thing I have to do is write an expander, say for @splicecode (bikeshedding welcome 😄), in src/Expanders.jl? For this I need the the abstract types, the Selector.order, Selector.matcher, and Selector.runner.

2. What should it do? Ie if I have the code snippet read (as as string code) and the language parsed (lang, defaults to "julia"), what should the Selectors.runner insert? OK if I make a

   page.mapping[x] = Markdown.MD("```$lang\n$code```\n)

   or is it something more complex?

Sorry if the questions are trivial, this is my first time looking at the code of Documenter.jl.

[mortenpi]

Does it enable syntax color highlighting for Julia ?

You can always add the language to the code block, e.g.:

````@eval
Markdown.parse("""
```julia
$(readstring("PATH_TO_FILE"))

""")

@tpapp: Awesome! Yes, an expander is what is needed.

The other expanders often construct specialized *Node objects, in which case the rendering code can decide to do something more sophisticated with it. But just splicing in an ordinary code block is probably enough here.

The mapped object should be a Markdown.Code(lang, code) though.

For the interface, I would personally probably lean towards something similar to @meta. I.e. the user gives the file name with key/value pairs. This way it can be expanded in the future, if need be. In the following example, File would be mandatory, Language optional (would allow the user to specify the language explicitly; the default should either be Julia or determined from the extension).

```@splicecode
File = "..."`
Language = "julia"

A bit of thought is needed for how the file path will be resolved. Should we assume that the spliced files are always external to the `src/` directory? In that case, whatever we go with, it should be consistent with #552, I think. What we settled on there is to have a new [`makedocs` option called `external_root`](https://github.com/JuliaDocs/Documenter.jl/pull/552#discussion_r148958778).

[tpapp]

@mortenpi: what I was thinking of is a syntax

```@splicecode
file = "..."                   # mandatory
lang = "julia"                 # optional
lines = 1:end                  # optional, intepreted within the range selected below
after = "# CODE BLOCK 1 START" # optional, mark the beginning of a code block
before = "# CODE BLOCK 1 END"  # optional, mark the end of a code block

so that the user could select based on lines and also strings marking some code snippet.

My understanding is that I can use `Utilities.parseblock` to get these and eval the right hand sides, is this correct? Is there any validation mechanism, so that
```julia
file = run(`rm -Rf ~/`)

is excluded? Or is it just plain eval?

What I am unclear about is external, should

file = "../../src/include_this.jl"

and similar not take care of this? How can I use external in this context?

[mortenpi]

LGTM @tpapp! I would stick to CamelCase with the keys though, to be consistent with the other at-blocks.

parseblock just parse()s into AST and does not eval, as far as I can tell.

Regarding external.. yes, you are right I think.. by default the path should be considered relative to the current .md file (or .jl file containing the docstring with the @splicecode block). File = external("...") could be something useful for the user though, so you could have File = external("examples/sth.jl"), instead of File = "../../examples/sth.jl".

[cossio]

I was looking for this functionality. Was there a PR made in the end?

[tpapp]

@cossio: I didn't follow up on this. FWIW, I don't think that source code should be lifted into documentation like this in most cases. For literate programming, we have Literate.jl.

[cossio]

Thanks. Where can I look at some examples of Literate + Documenter interaction? Or Weave + Documenter?

[tpapp]

https://fredrikekre.github.io/Literate.jl/v2/documenter/

[cossio]

Oh yes I had read that, thanks. But what I meant was how to generate the pages automatically during CI. Like, what should I put in the make.jl file of Documenter to make it generate also the Literate pages and have them appear linked in the documentation site? Probably there are some packages out there doing this already, I would just like to inspect their source code to see how they do it.

[odow]

I think this issue could be closed. There is very little reason to include code snippets from source (no people asking for 5 years), and we have Literate.jl which is serves the purpose of embedding examples etc.

There's an open issue for explaining how to use Literate: https://github.com/JuliaDocs/Documenter.jl/issues/2200

[mortenpi]

Actually, let me reopen this: let's refocus this issue around the splicing feature: https://github.com/JuliaDocs/Documenter.jl/issues/499#issuecomment-346994594

I actually do have the need for something like that. Although, I think it should be a plugin package in first instance.

[leios]

I realize this is an old discussion now and there is more in #2456, but I'll mention a few use-cases where this would be quite useful:

1. Many packages have extensive examples that exist in the github repo and should also be discussed in more detail in the docs. As it is right now, we need to either duplicate the code (which means updating the code 2x if there is an API change in the library) or rewrite the examples into a Literate file, which means adding a lot of hefty markdown to relatively small files. Here, it's important to be able to import from specific lines in each file, so you can explain each code block in the docs thoroughly. I think the @splicecode macro proposed above could work for this use case.
2. The JuliaGPU ecosystem (for example) has a few different APIs for different hardware (AMD, CUDA, Metal, OneAPI, OpenCL, KernelAbstractions, Vulkan, etc). It would be nice to be able to replicate the CUDA.jl documentation, but for all APIs. You could (for example), create a simple UI element that shows all the different APIs above each code block (or at the top of the page) and allow users to click to select which one is relevant to them

I have been waiting for this feature for a while so I can port the https://algorithm-archive.org over to Documenter. There, we have syntax to switch between code blocks that looks like this:

{% method %}
{% sample lang="jl" %}
Some example text here introducing the block with Julia-specific stuff
[import:2-7, lang:"julia"](code/julia/monte_carlo.jl)
{% sample lang="clj" %}
Similar text block for clojure
[import:3-10, lang:"clojure"](code/clojure/monte_carlo.clj)
... Other langs ...
{% endmethod %}

This obviously includes more than just splicing in code, but also a way to flip between different APIs. So maybe an additional @swapblock macro on top of the @splicecode?

Had anyone started development of such a plugin?