[Off-topic] Discussion about similar workflow as vimtex for compiling/knitting R-markdown in vim

[krishnakumarg1984]

I very well understand that this issue is entirely off-topic and needs to be closed. Instead of sending the author a private email, I chose to submit an 'issue' here:

• to gather responses from the wider community who may be having similar needs
• to serve as a reference for posterity to those searching online for a similar workflow

Also, apologies that the issue description below is more of a narrative setting the scene rather than being purely technical.

vimtex workflow is so natural, and I find this to be an indispensable tool for journal/conference articles (as well as my PhD thesis). However, in most of these occasions the decision-making lies with me and/or a small eclectic group of like-minded individuals who are technically inclined.
When required to work with an established workflow in a company (think heavily branded template headers for PowerPoint presentations and Microsoft Word documents), I have realised the hard way that fighting these will only lead to needless difficulties both from technical and social perspectives. On the other hand, giving up plain-text goodness and document versioning and vim that people spend so much time learning and customising does not sound attractive.

A reasonable answer to the above issue is pandoc and the automation wrappers around it such as pandocomatic. While this does cover significant ground, it does not take one all the way. I have narrowed it down to two reasons:

• LaTeX's markup syntax is heavier and detracts the authors of documents who not familiar with it
• pandoc's conversions requires lots of manual intervention, which doesn't really scale well (especially in collaborating on a long-running project with a non-latex user)

Furthermore, there are a few issues. Tex's most common output format (PDF) is designed for a fixed static screen wherein the absolute placement of glyphs on pages are precisely calculated using an optimization algorithm (box-glue-penalty model). All the HTML converters I have tried (HeVeA, latexml, lwarp, tex4ht/make4ht etc) do not give acceptable results for documents of moderate complexity. Furthermore, I haven't had success with generating display-aware reflowable HTML.

The paradigm I'd like to be seamlessly automatic is write once and obtain multiple outputs. To this end, the starting input format makes a difference in the final output quality, and certainly a heavily customized latex document is not going to be easy to convert and collaborate (with a non-latex user). To this end, I investigated

• gitbook
• asciidoc
• rmarkdown
• madoko
• doconce

Note that some of these are formats, while the others are tools. It certainly appears that using amarkdown-like syntax can cover maybe 80% of the standard typesetting aspects. Of course, the power of LaTeX is a poor trade-off for an oversimplified markup language like markdown, but perhaps this is what it takes to integrate smoothly with colleagues' workflow.

In the end, I have shortlisted R markdown which makes dynamic reporting (slides,reports,articles,books etc. with computations embedded) easy and already builds on existing tools (pandoc + knitr). However, the document compilation happens inside an 'R' environment with a command.

I am looking for a workflow similar to vimtex such as: \ll for continuous compilation (compile upon save), \le for populating quickfix error window, \lv to view output (The output formats are indicated in the metadata block). The closest I came to a usable plugin is vim-rmarkdown which provides a command :RMarkdown[!] [OUTPUT_TYPE] [- RENDER_ARGS[ -]] [OUTPUT_TYPE_ARGS], but it is tedious to invoke this manually upon every save.

It would be great if some sort of vimtex workflow can be proposed by the community here. Again, personally I intend this to supplement vimtex and not supplant it for my own documents.

[lervag]

Thanks, @krishnakumarg1984, I think you're post is very interesting and it "strikes a nerve" with me. I'll take time to give my thoughts later when I have more time, but I wanted to ping some more people I think may have some ideas/thoughts on this topic: @jabirali @kiryph @clason

[clason]

Well, I don't think I have anywhere near as much to offer on this as @lervag, but here are two comments:

1. if you want conversion on save, you can set an autocommand in your vimrc:

   au BufWritePost *.Rmd RMarkdown

   (assuming *.Rmd is the extension of your R markdown files)

2. Rather than start from scratch, I would recommend using vim-rmarkdown (if it's at all usable in principle) and making (and helping at least with testing) feature requests to support your workflow. Open source development (after the initial "scratch your own itch" period) is driven by (reasonable/reasoned) suggestions from users. This way you help making the plugin better for everyone! (If the author of vim-rmarkdown does not want to go in that direction, there's no sense in pressuring them, of course.)

[lervag]

When required to work with an established workflow in a company (think heavily branded template headers for PowerPoint presentations and Microsoft Word documents), I have realised the hard way that fighting these will only lead to needless difficulties both from technical and social perspectives. On the other hand, giving up plain-text goodness and document versioning and vim that people spend so much time learning and customising does not sound attractive.

Agreed (and I would assume most of use Vim users would agree)!

pandoc's conversions requires lots of manual intervention, which doesn't really scale well (especially in collaborating on a long-running project with a non-latex user)

Also agreed. I've never really tried to collaborate with others on MS Office documents by converting back and forth with pandoc (or other tools). My current workflow is simply to "accept defeat". :\

The paradigm I'd like to be seamlessly automatic is write once and obtain multiple outputs.

Do I understand that you are mostly interested in a "one way" collaboration? I.e. that you write something that you can share as an MS Office file (or whatever)? You are not looking for a solution "both ways", e.g. where your colleagues send you a document that you are asked to work on?

To this end, the starting input format makes a difference in the final output quality, and certainly a heavily customized latex document is not going to be easy to convert and collaborate (with a non-latex user).

Agreed.

Of course, the power of LaTeX is a poor trade-off for an oversimplified markup language like markdown, but perhaps this is what it takes to integrate smoothly with colleagues' workflow.

Again, do I understand that your goal is now to write something that you have full control over, except that you want the output to be some format your colleagues approve of?

In the end, I have shortlisted R markdown which makes dynamic reporting (slides,reports,articles,books etc. with computations embedded) easy and already builds on existing tools (pandoc + knitr). However, the document compilation happens inside an 'R' environment with a command.

I've also found R markdown interesting for the exact same reasons. I have to admit I never found the time to really learn to use it. I've taken note of vim-rmarkdown, but, again, I've not installed it or used it yet.

It would be great if some sort of vimtex workflow can be proposed by the community here. Again, personally I intend this to supplement vimtex and not supplant it for my own documents.

My initial thought is that it seems sensible to reach out to the vim-rmarkdown devs. You could open an issue there and see if it gets some traction. Feel free to link to this thread as well.

Still, though: My impression is that the project is not very actively maintained (there's 17 open issues and 2 open PRs, and the last commit was made last August). Thus, it would seem like there is room for a fork or a new similar type of project. I like this kind of work, but I have to admit I don't have the time to address it myself. I guess this resonates with @clason's second comment.

[jabirali]

I have also been drafting some documents in MarkDown or LaTeX at work and Pandoc'ing to Word :). That's always been a unidirectional conversion though, with all subsequent revisions done in Word...

I don't have much experience with RMarkDown though, but I'll try to chime in as well. Regarding the continuous compilation (\ll) command @krishnakumarg1984 mentioned, I'm wondering if this wouldn't fit better in a dedicated tool? For instance, vimtex can use latexmk as a compiler backend. This is a smart tool that in continuous mode (-pvc) monitors all relevant files, and runs compilation commands (pdflatex, bibtex, etc.) as required. Being a standalone tool, it can be used as a backend for Vim, Emacs, TeXMaker, etc., or it can be used by itself (in a shell or from scripts).

Calling RMarkDown with an autocmd as @clason suggested seems like a good solution if you just edit one giant *.Rmd file. But assuming that RMarkDown files can include other files (e.g. standardalone *.R files, *.svg figures drawn in Inkscape, *.bib generated by Zotero/Mendeley), I would personally want a recompilation every time one of those included files are modified too. I think a good solution would be to have a separate latexmk-like utility that analyzes what files are included by your *.Rmd, monitors all those files for changes using e.g. entr or inotify(7), and runs appropriate recompilation commands automatically. This tool can then be called from plugins for Vim, Emacs, etc., resulting in less duplicated effort among editors for supporting the same workflow. And I suspect it would scale better to handling large projects (e.g. a book consisting of many files).

Sample implementation: use a set of regexp's for parsing where a *.Rmd master file includes other files; recursively parse *.Rmd, *.R, *.md files to discover more included files that the document depends on; run entr on all the discovered files; call pandoc + knitr when entr notices change. Ensure (via command-line options to pandoc and knitr if needed) that potential errors are spit out in a format that can easily be used to populate the quickfix from the vim side. When typing the equivalent of \ll, the main job of a vim plugin would then be to start this dedicated tool as an async process, and listen for error messages coming back from the continuous compilation.

What do you think?

[krishnakumarg1984]

Also agreed. I've never really tried to collaborate with others on MS Office documents by converting back and forth with pandoc (or other tools). My current workflow is simply to "accept defeat". :\

Yes. And I have to acknowledge that for those with non-development background (managers, industry colleagues, PhD supervisors amid others), the well-integrated track changes feature in MS Word is the single feature that enforces this workflow. The argument that plain-text can potentially be a better track-changes does not really fly well for the following reasons:

1. (git) diffs/merges etc are quite a technical concept
2. To really work well, they'll probably need to be one sentence per line without hard-breaks
3. Seamless integration of MS Word with the company's MS Teams' server (can see real-time edits by other team members without any exclusive locking, integration with LDAP server (Microsoft AD) and deep integration of Office 365 inside Outlook as well as inside the browser). You can literally view/edit the rendered version of the document in lots of places simultaneously including mobile phones. Role-Based access control and easy lookup of contacts, author metadata etc. etc.

The collaborative experience cannot simply be beaten (while the authoring experience takes a hit for plain-text/vim aficionados). While Libreoffice/Abiword have improved a lot, they are still no match for non-ODT documents to Microsoft and anyway the LDAP/MS-Teams integration aspect is missing. So, we have to go all-out on the Microsoft Way (which is a pain for linux users since are forced to use the browser version of office365 for writing docs)

The paradigm I'd like to be seamlessly automatic is write once and obtain multiple outputs.

Do I understand that you are mostly interested in a "one way" collaboration? I.e. that you write something that you can share as an MS Office file (or whatever)? You are not looking for a solution "both ways", e.g. where your colleagues send you a document that you are asked to work on?

No. Actually, I am thinking two-way collaboration for a particular type of doc, and is feasible to do so with somewhat technically minded folks with a lightweight markup language. Let me explain.

Although documents can span a whole continuous spectrum (and I am not naming each individual type here), I am going to oversimplify the truth and just consider maybe in an engineering company that some docs are traditional product reports/ application notes etc (text heavy paragraphs, some figures, ugly tables, cover pages with company branding etc.) while others especially with a software-twist are a bit more workflow oriented intermixing code and text i.e. literate programming (think jupyter notebooks). The 1st category of docs can probably never become a candidate for plain-text within company cultures (unless you mind becoming socially ostracized), which leaves us with the 2nd category. But jupyter notebooks themselves don't make for good reports. Though we can argue that they can be converted and exported to some output formats, the reporting still seems err. not professional, and looks like a lab notebook (rather than customer-ready/publication-ready work). I have seen duplication of effort wherein contents from finalised notebooks are copied to MS Word docs (ouch!). Also, content can only be evaluated within a code block (and not within a markdown block), so you can't have inline evaluation of symbolic or numerical maths. Additionally, jupyter notebooks wont' work well with version control.

This is where R-markdown in conjunction with jupytext totally helps. Stands on shoulders of giants

• Can write in plain-text and inline/displayed evaluation/presentation of code, math and text is no problem
• A markdown-flavour still stands a chance to convince somewhat technically minded colleagues to atleast give it a try since the entry barrier is low
• lots of coding languages supported out of the box (something like pythontex, but many times more). I am also excited about using symbolic math inline.
• Bi-directional conversion and syncing of jupyter notebooks and R-markdown is facilitated by jupytext (which is a Godsend)

I evaluated a lot of options rigorously and decided that R-markdown is indeed the best available (and together with its extensions such as bookdown, thesisdown etc.) seems to be the most well-rounded.

Of course, the power of LaTeX is a poor trade-off for an oversimplified markup language like markdown, but perhaps this is what it takes to integrate smoothly with colleagues' workflow.

Again, do I understand that your goal is now to write something that you have full control over, except that you want the output to be some format your colleagues approve of?

No. As explained above, I think R-Markdown has the strong potential to entice some technical-minded colleagues to using it (even if they use Jupyter) for some type of documents which has some computation outputs (especially inline).

Still, though: My impression is that the project is not very actively maintained (there's 17 open issues and 2 open PRs, and the last commit was made last August). Thus, it would seem like there is room for a fork or a new similar type of project. I like this kind of work, but I have to admit I don't have the time to address it myself. I guess this resonates with @clason's second comment.

Well. As @clason observed, it is better to stand on the shoulders of giants and build upon existing work (which is exactly what R-markdown does (markdown-flavour with extensions + pandoc + knitr). But vim-rmarkdown probably isn't the right solution yet (and is nowhere near the polish levels), but is maybe a good starting point for syntax?

I'd like to slowly emulate the features of vimtex if such an analogy is even possible. I have been perpetually learning python, and this sounds like a good use-case to put some learning to real-world use using the python3 standard library functions. I have no experience writing vim plugins, and of course I'd like to leverage your help if possible.

For example, initially we could parse document metadata from the yaml frontmatter, and if latex/pdf is one of the output targets, we could hand-over control to latexmk (or whichever tool is specified in a user config file). I am not sure, just brainstorming here.

[clason]

I can't resist adding a few cents more:

1. For collaborative "Word-like" editing, there's also https://www.onlyoffice.com (integrated in OwnCloud and possibly NextCloud). But you're probably trying to get away from "Word-like" altogether...

2. For collaborative rich markdown editing (albeit without R or Python integration), there's https://github.com/hackmdio/codimd, which sounds like it would actually fit your workflow fairly well.

3. If you're OK with using neovim instead of vim, this has Lua as a first-class scripting language, which may be easier (and more pleasant) to work with for people coming fresh to plugin development (see, e.g., https://teukka.tech/luanvim.html) -- not to mention being potentially significantly more performant...

4. And to cut to the chase, what you seem to be asking @lervag -- in so many words -- is whether it wouldn't be possible to extract vimtex's compiler infrastructure into a general-purpose framework that you could then use for RmarkDown (or pandoc, or what not) ;)

[krishnakumarg1984]

@clason If editing docx files directly, there is absolutely no substitute to MS Word (not because it is great, but because of its deep integration with Office 365, Share-point/Teams and real-time collaboration). The Word use-case is well beyond the actual product itself, it is about the whole ecosystem (access to LDAP directory names etc). These days, in the track changes comments, you can even @ prepend and a whole list of user full names will drop down (sorted 1st by your local organisational unit/team followed by all other members within the whole company). You can select a person so far not involved in the document for collaborating (and you don't even have to know their email id), and they will get a notification email using LDAP. In fact, an entire business vcard opens up showing their office location, telephone extension, their manager's name and email and their place in a organisation's hierarchy). In a Teams' virtual meeting, you could bring it up seamlessly within that app, edit it and things are autosaved, while a collaborator could work on it later using a full copy of MS Word later on, and so on and so forth.

In short, there is no point looking for Word replacement when dealing with companies using O365 subscription. MS Word is beyond that mediocre single product (albeit one with an underwhelming editing experience and output quality).

For collaborative rich markdown editing (albeit without R or Python integration), there's https://github.com/hackmdio/codimd, which sounds like it would actually fit your workflow fairly well.

Thank you. This seems useful. But it is not vim. There are literally 50 editors/IDEs for LaTeX. Yet, the users here use vim for many specific reasons (eg. piping through external utilities and not just keybindings, working inside terminal and a few other). For me, anything plain-text should be written in vim.

If you're OK with using neovim instead of vim, this has Lua as a first-class scripting language, which may be easier (and more pleasant) to work with for people coming fresh to plugin development (see, e.g., https://teukka.tech/luanvim.html) -- not to mention being potentially significantly more performant...

plugin development...hmm..big words... I haven't exactly said plugin development, except just teasing out user opinion of experts here ;) . I am doing this only because vim + technical documentation is a niche intersection and I don't know if any subreddit or stackexchange community exists for this specific intersection of interests. Plus I was specifically interested in vimtex's workflow and concepts.

I'll have to learn yet another language lua (albeit a tiny one). Doesn't python provide a more powerful interface? Plus looks better on CV for a data-centred job application.

And to cut to the chase, what you seem to be asking @lervag -- in so many words -- is whether it wouldn't be possible to extract vimtex's compiler infrastructure into a general-purpose framework that you could then use for RmarkDown (or pandoc, or what not) ;)

I really wasn't sure what I was asking. I still don't have a question in mind. I don't know how this can be done. What I want is a "knit" button like R-studio, but with all the text-objects, motions etc and callable from within vim like vimtex. vimtex is written in vimscript. Even if it's core is refactored as a library, I don't personally know how I can leverage it. Plus, rmarkdown has so many other output formats, which vimtex doesn't cover obviously. In short, I don't know what exactly might be the suitable pipelines/backends/details. I just know what I want at a high level - that I'd like to edit, compile, fix errors and view output documents from the comfort of vim.

[lervag]

Sorry about the late reply! Too much text to cover and too little time, so I've postponed for quite some time...

... So, we have to go all-out on the Microsoft Way (which is a pain for linux users since are forced to use the browser version of office365 for writing docs)

Yes, I think you are quite right. Unfortunately. :\

I evaluated a lot of options rigorously and decided that R-markdown is indeed the best available (and together with its extensions such as bookdown, thesisdown etc.) seems to be the most well-rounded.

Thank you for the nice writeup, it is educative for me and seems to give a clear indication that I should take the time to learn more about R markdown! And jupytext, which I also did not know about.

I would be happy if you could share an example or two, or some simple writeup about a workflow of yours in this regard?

I'd like to slowly emulate the features of vimtex if such an analogy is even possible. I have been perpetually learning python, and this sounds like a good use-case to put some learning to real-world use using the python3 standard library functions. I have no experience writing vim plugins, and of course I'd like to leverage your help if possible.

I would be happy to help! It seems possible a win-win, in that I can simultaneously learn to write my documents in R markdown and leverage more literate programming in my documents and notes.

I really wasn't sure what I was asking. I still don't have a question in mind. I don't know how this can be done. What I want is a "knit" button like R-studio, but with all the text-objects, motions etc and callable from within vim like vimtex. vimtex is written in vimscript. Even if it's core is refactored as a library, I don't personally know how I can leverage it. Plus, rmarkdown has so many other output formats, which vimtex doesn't cover obviously. In short, I don't know what exactly might be the suitable pipelines/backends/details. I just know what I want at a high level - that I'd like to edit, compile, fix errors and view output documents from the comfort of vim.

I think this is partly the way to "start" a plugin. That is, become aware of your own workflows. In particular, I think we are discussing two things:

• Compilation: One approach could be to rely on the compiler backend in Vim where you define the makeprg and errorformat. Another approach could be to simply use a Makefile and the terminal. Personally, I tend to start here, and then bring it to Vim if I see some real benefits of doing that. Having it in a Makefile (or script) is also a good way of describing the exact workflow from text file to compiled output.
• Text objects: If you want a lot of LaTeX/Math in your markdown documents, then it could be possible to copy or rely on some of vimtex's definitions. One could load vimtex text objects also for markdown files, although it is not "supported".

I also think what we are discussing is closely related to Emacs's org mode. It seems to support almost everything you want, except it is not Vim. Still, with Evil mode and such, I have the impression that Emacs can take you far.

---

In any case, I'll close this issue now. Feel free to respond and continue the thread, but it is clearly "off topic" for vimtex specifically.

[lervag]

Thanks also to both @jabirali and @clason for adding thoughtful comments to the topic.