Proposal to add support for BibTeX citations and references

[ali-ramadhan]

Hi Documenter devs,

A few of us would love to see support for in-text citations and reference generation in Documenter through BiBTeX (see https://github.com/climate-machine/CLIMA/issues/152 and https://github.com/climate-machine/Oceananigans.jl/pull/474).

I guess I envision being able to pass a BibTeX file to Documenter and automatically be able to cite references in the text with something like [<label>](@ref) (which would pull the info from the BibTeX file), and also generate a bibliography from the BibTex file through something like

```@bibliography

I'm wondering if this is something that could be worked on by someone unfamiliar with Documenter. Thought it might be good to open an issue first to see if it's possible and if I'm missing anything. Not sure of the best approach but here's what I've gathered so far.

It looks like there was an attempt back in 2017 by @lucianolorenti to add a `bibtex` option to `makedocs` that allowed citations with `[<label>](@ref)`: https://github.com/JuliaDocs/Documenter.jl/issues/379#issuecomment-292157714

But it had to use the Python package `pybtex` to parse bibtex files. But since then there's been a native Julia parser: https://github.com/JuliaTeX/BibTeX.jl

So maybe if we can get that `bibtex` option to work with BibTeX.jl then we can pretty easily have citations in Documenter?

But then I came across a recent issue (https://github.com/JuliaDocs/Documenter.jl/issues/1018) where @mortenpi suggested that proper bibliography support should be done via a plugin like [Citations.jl](https://github.com/adamslc/Citations.jl), although it was mentioned on Slack that it's an unmaintained package that would need some work to bring it up to speed with the current version of Documenter and it would have to be registered.

It seems like the best approach would be to polish off Citations.jl to work with Documenter v0.24 and register it? Would https://github.com/JuliaDocs/Documenter.jl/issues/745 have to be resolved first?

cc @glwagner @charleskawczynski @simonbyrne who might be interested as well.

[mortenpi]

Yes, I do think a plugin package is a way to go and starting from Citations is probably a good start. Among other things, it would be nice if core Documenter wouldn't depend on things like BibTeX.jl.

• For the reference links, I think a custom [](@cite) link would probably better? I am not sure it makes sense to overload at-refs for this purpose, at least if you want e.g. numbered superscripts etc. Although maybe something like [random text here](@ref cite"citationindex") could be an option to link to the bibliography within the text.

• 745 is unrelated, as it would be up to the plugin to generate the correct HTML, which can contain all the necessary anchors.

• Documenter's internals do not have a super thought out design for plugin packages currently (but it would be great if it did). So, as an MVP here, I would just try to hack something together that overloads some functions in Documenter to make everything work (note that all the build pipelines etc. are designed to be extended).

  If it is bound to a particular 0.x release of Documenter, it is unlikely to break. And then we can figure out what long term public API we want to provide so that the package would work across Documenter versions.

I won't have time to work on this feature myself really, I would be happy to brainstorm this here or on Slack. I can give a hand with whatever refactoring might be necessary for Documenter's internals though.

[lucianolorenti]

Hi! A while ago I ported most of pybtex to Julia to allow markdown, latex and plain text output from a bib file. The package is BibTeXFormat.jl I updated the code to work in Julia 1.2 and to use the new official bibtex parser. Some output can be seen in https://lucianolorenti.github.io/BibTeXFormat.jl/latest/#Markdown-example-1 Some tests still fail because I could not find a package like pylatexenc in Julia that allows me to convert between things like {\'e} to é.

The package still needs some work and I don't have much time to devote to this, but perhaps, it can be useful.

[simonbyrne]

I could not find a package like pylatexenc in Julia that allows me to convert between things like {\'e} to é.

Sounds like a great idea for a GSOC project?

[simonbyrne]

Or possibly a Code-In project, if someone was willing to provide some guidance?

[fredrikekre]

Maybe it exists in Latexify.jl? If not maybe the functionality would fit there? Not sure.

[ali-ramadhan]

Sorry to let this issue sit around for so long but thanks to @Azzaare and his new Bibliography.jl package I think I was able to get a minimal working example of hacking in Bibliography.jl as a Documenter.jl "plugin" without too much work!

I’ve put up the code up on https://github.com/ali-ramadhan/DocumenterBibliographyTest.jl but it's a little messy (BIBLIOGRAPHY is a global const for example) so @mortenpi suggested posting here to discuss how it can be improved. I'm also not sure where this code should be placed or if it should be part of an existing package. Perhaps if it's truly a plugin then it should be it's own package?

Too lazy to set up documentation builds but here's a screenshot of a page with two references:

@article{ahu61,
   author={Arrow, Kenneth J. and Leonid Hurwicz and Hirofumi Uzawa},
   title={Constraint qualifications in maximization problems},
   journal={Naval Research Logistics Quarterly},
   volume={8},
   year=1961,
   pages={175-191},
   doi={10.1002/nav.3800080206}
}

@book{ab94,
   author = {Charalambos D. Aliprantis and Kim C. Border},
   year = {1994},
   title = {Infinite Dimensional Analysis},
   publisher = {Springer},
   address = {Berlin},
   url = {https://www.springer.com/gp/book/9783540295860}
}

# DocumenterBibliographyTest.jl documentation

Hi!

In this page I will cite an article [ahu61](@cite) and a book [ab94](@cite).

# References

```@bibliography```

The idea is that you read in a .bib file with Bibliography.jl (which I called const BIBLIOGRAPHY) and the new @bibliography expands to print all the references in that file formatted with URL/DOI links in an HTML description list, and each reference gets an anchor added. Then whenever you use the @cite URL syntax, it replaces the URL with an author-year citation that links to the anchor in the bibliography. So hopefully this means that the bibliography can be placed on any page and citations should work from any page as well.

Limitations: 1 .bib file only, and @bibliography prints all entries even references that were not cited (but these are minor I think). Also https://github.com/Azzaare/Bibliography.jl/issues/3.

[kellertuer]

Thank you for your work, this is what I was hoping for in Markdown code :)

I wouldn't mind having this within documenter directly, though it adds Bibliography.jl as a dependency.

Just for the limitations I think one of those should be resolved, I can either have one large bib-file (which is what I prefer) and a selection printed or a bib-file per page for example and printing all. Would it be possible to print only cited ones? That would be great.

edit: another possibility is of course to have the bibliography on its own page (a bibliography.md with just one line) but to the @cites then link between pages? Might be difficult, too, so I might still prefer the per page selected variant.

[ali-ramadhan]

@kellertuer I updated the test repo with a bibliography on its own page (references.md) and the @cite commands work on other pages which is nice. That was my use case as well: have one big .bib file and have the references listed out on a separate page with citations peppered throughout the docs.

Regarding the limitations, I don't know enough about Documenter.jl to implement printing only cited references. The @bibliography is expanded first as it is part of ExpandersPipeline, before the @cite commands are processed which are part of DocumentPipeline, which is neccessary to obtain the anchors/links for each reference. So when expanding the bibliography, we have no knowledge of what has been cited so far.

I think it shouldn't be too hard to add support for multiple bibliographies but I'm not sure which approach would be best. We can write e.g. @bibliography books.bib to expand one .bib file but then the @cite command would require access to all bibliographies. There must be a way of implementing this.

For a very first version of this plugin, allowing only one .bib file without worrying about whether the reference was cited may be reasonable just to figure out the design of the plugin before adding more features.

[kellertuer]

Cool, so then one of my approaches would work!

I think I prefer the one with only-cited ones, but you are right, it might b worth first checking for the plugin and its design and then (maybe with the help from more experienced Documenter programmers) extend the plugin to the two other features.

[mortenpi]

This is cool @ali-ramadhan! Just a few thoughts from me:

• Yes, a separate plugin package would be preferred.

• You can pass objects that are subtypes of Plugin to makedocs as positional arguments. They get stored in the .plugins field: https://github.com/JuliaDocs/Documenter.jl/blob/3e2090abf8064c55ac8828c060513de7f8e6959f/src/Documents.jl#L260-L265

  With that you can you should be able to store the bibliography without relying on globals.

• Although re-using at-raw blocks is a good hack to get started, I think the generation of HTML would ideally be done on a per-writer basis. You could create a new type (e.g. BibliographyNode) that you can drop into page.mapping. Then each writer can implement how they want to handle it. For HTML, I think a domify method (e.g. like the one for DocsNode) that would produce a DOM node should be sufficient.

  This could also get around the problem with not knowing what was cited -- domify gets called in the last build step. expand_citation should probably just push something to the bibliography plugin that you could then cross-check as you render.

The internal APIs, that I can see, that we'd need to formalize:

• Assigning to page.mapping.
• Documents.walk
• Creating anchors with Anchors
• Error handling (push!(doc.internal.errors, :citations))
• Utilities.locrepr
• Possibly: DOM and domify

[mortenpi]

The plugin exists now: https://github.com/JuliaDocs/DocumenterCitations.jl

[kellertuer]

It not only exists, it also works really great! 👍