Adding a Julia engine for knitr?

[bschneidr]

The JuliaConnectoR package is currently the most stable interface to Julia from R, not only because of the TCP-based design but also because the main alternative, JuliaCall, seems to be unstable and not really actively maintained. I reached out to the developers of knitr about the idea of using JuliaConnectoR instead of JuliaCall when running Julia code. They suggested reaching out and seeing if you, the package authors, would be interested in adding a knitr engine similar to JuliaCall, so that JuliaConnectoR would be used instead.

Here's the relevant discussion thread in the knitr repo: https://github.com/yihui/knitr/issues/2295#issuecomment-1952024524

Does this seem like something you'd be interested in supporting? I think @cderv and other Posit/knitr developers might be able to help implement this if this is something you'd consider adding to the package.

The reason I reached out to the knitr team is because I recently wrote a Quarto blog post about using R+Julia and the JuliaConnectoR package. The JuliaConnectoR code worked super smoothly, but I had a really frustrating experience trying to get Julia to run in Quarto using the JuliaCall knitr engine.

[stefan-m-lenz]

It is nice to hear that the JuliaConnectoR package is useful, thank you for your feedback. Thank you also for sharing your blog post!

About the state of the JuliaConnectoR: I have to say that the package is currently only maintained by myself. I don't have the capacity to add bigger new features to the package. Fortunately, with the loose coupling of Julia and R via TCP, there is not much work to do to keep the package working also with new Julia versions that come out. So I don't see problems with maintaining the package further and keeping it working.

That being said: What you describe could definitely be done. Instead of putting the functionality in the JuliaConnectoR package, I would recommend to create a new R package for it, however. This new package (e.g. "JuliaKnitR") could have the JuliaConnectoR as dependency. It can contain the Julia code and the R code that have the necessary transaction logic for supporting knitr (e.g. transferring images) and rely on the JuliaConnectoR to communicate with Julia.

I think this modular approach is better. It has the following advantages:

• It reduces the burden on me as a maintainer of the JuliaConnectoR package and shifts the responsibility. As you can see in the JuliaCall code, there are some special hacks for different plotting packages (E.g. https://github.com/Non-Contradiction/JuliaCall/blob/c05473bea78a0197c639f7e82ab1c6f2e943e1cc/R/zzz.R#L228) . Having to deal with many such specific requests would overburden me.
• The logic that is necessary for maintaining the knitr support is bundled in the new package and thus more easily understandable and maintainable. I took a look at the JuliaCall code and I found the code related to knitr scattered over the package. It is unclear to me, for example, whether the aforementioned "dirty fix for Plots GR backend" is necessary for knitr or for something else.
• Testing is faster: The JuliaConnectoR package has already an extensive test suite which makes sure that the data is transferred correctly and that the behaviour stays the same between different Julia and R versions. Adding more tests for knitr would make it hard to run all the tests, slow down development, and waste energy unnecessarily.
• The backend could be switched easily later. If another package will be better suited than the JuliaConnectoR in the future, the "Julia backend" could be switched easily because the logic for knitr is already separated.

I would be happy to give my advice when starting a new package for the knitr support. We can also discuss this further here.

[stefan-m-lenz]

@bschneidr @cderv I've created a fork of knitr: https://github.com/stefan-m-lenz/knitr There I've changed the eng_julia function in engine.R and added a file KnitrEngine.jl in the inst/misc folder. Using Julia in a Quarto notebook works now already with textual output. I haven't implemented plots yet. I don't understand yet how that works.

In the eng_tikz function I find the following lines

  options$fig.num = 1L; options$fig.cur = 1L
  extra = run_hook_plot(fig, options)
  engine_output(options, options$code, '', extra)

(see https://github.com/stefan-m-lenz/knitr/blob/ea72f409cd794d4df176dc8d5d3b27b905eb2dfc/R/engine.R#L405)

Could you explain that? I guess I have to do something similar to implement plots for Julia.

Another question: How do I get the information about the size of the textual output (number of columns) or the graphical output (width in pixel/cm/inch). Is this usually an option element? How is it implemented normally? What are good defaults for that?

When I look at the code I've written to implement Julia in knitr, I don't think it is very intrusive and maybe the best solution would be to integrate this directly into knitr and not making a separate package. Would this be an option for you?

My plan to implement plots would be to look in the Julia KnitrEngine.run function whether the result is a plot, save the plot to a file in Julia and return the file link (file://...) as a string.