Transition from org-ref version 2 to org-ref version 3 - roadmap

[jkitchin]

I originally thought that the new citation syntax in org-mode was going to make the citation links in org-ref obsolete. Now that I have worked my way through org-ref-cite (https://github.com/jkitchin/org-ref-cite), I no longer think that is true. It is more clear to me these two projects had different goals, and those goals are not fully compatible. It comes down to org-ref using bibtex/biblatex as the predominant citation processor, and org-cite using CSL. These two processors have different syntaxes, and I don't think it is possible to have a single approach that works for both of them without making compromises in capability. I use org-ref professionally, and I don't want to make compromises on its functionality.

In org-ref, I used the link type to indicate the LaTeX export command explicitly, so it looks like what you would read about in LaTeX documentation. That is a feature IMO, so you don't have to figure out how to translate the cite style. Of course, that is only a benefit if you know LaTeX, and are aiming to generate LaTeX for scientific publications. The disadvantage of this approach is trying to define the export behavior for other backends, which has been limited in org-ref. org-cite, on the other hand, uses a generalized citation style like cite/style/variant which is the same across export backends. I found it was not easy to map these onto the full set of natbib/biblatex citation commands, and not easy to remember that mapping while writing.

Admittedly, the original syntax used in org-ref provided minimal support for pre/post notes in a clunky way that used the link description. The org-cite syntax supports this better, allowing common pre/postnotes, as well as individual pre/postnotes on multiple references. It turns out that org-ref can support this syntax natively in the existing citation links, which provides a path to exporting org-ref links to other backends using citeproc to render them.

Finally, org-cite and org itself do not provide adequate cross-referencing (or index/glossary functions) for scientific publishing, so it is clear that the rest of org-ref also still has a place in this ecosystem.

Soooo.... I have rewritten a lot of org-ref to support all these issues. There is a new "Version 3" in https://github.com/jkitchin/org-ref/tree/org-ref-3. I really struggled to decide this was the right thing for me to do; a lot of people have invested time in org-cite over the past years. In the end I decided to do it for these reasons:

1. This is the tool I need and use professionally on a daily basis. It has to do what I need the way I need it.
2. I (and others) have a lot of legacy documents that use org-ref, and it is clear I have to be able to support those for a long time
3. I thought it was worth pushing the link syntax to support this. The link behavior in org 9.0 was designed to support this kind of applications. I wish I would have figured out much earlier that this could all be done this way.
4. There is room for both approaches. They have different goals and applications.
5. I learned emacs-lisp to write org-ref, and the early code shows that. I am a better programmer now, and now the code-base is better than it was.

Version 3 supports the legacy link syntax, and a new link syntax that is identical to what is in org-cite.

I took this opportunity to clean up ~8 years of buildup. It now only supports bibtex-completion by @tmalsburg for candidates, opening notes, pdfs, light formatting, etc. The vanilla org-ref library is completion backend agnostic, so if you don't want ivy or helm, you won't get it, but you do have to wire in the completion backend you prefer (e.g. Selectrum, ido, etc.). I have removed the ivy and helm dependencies so you won't get them if you don't want them, although you will have to install them if you want them.

If you do like helm or ivy, you can load org-ref-ivy which depends on ivy-bibtex or org-ref-helm which depends on helm-bibtex. Good news on this front, these libraries will no longer mess with the order of actions in them; there are dedicated insert commands that look like those functions, but aren't them.

I removed all the org-ref-* variables that were just shadowing bibtex-completion-* variables to keep things simpler. I also took out all the key-bindings, in case you have different ideas. Those have to be set in your own init files now.

Apologies in advance to anyone who contributed code that is now gone. The code is still in version 2, and it can probably be added back, let me know what it is, and I can work it back in.

There are still some minor issues of over-active fontification between org-ref and org-cite that have not been resolved, but they only affect the cite: link.

I am going to use "Version 3" myself for a few more weeks to get it as bug-free as I can before I merge it on the master branch, at which point it will become the standard version available in MELPA. It is pretty functional at this point, and I only anticipate fixing some small corner case bugs in the coming weeks. I still need to test it more thoroughly in the new version of org-mode with org-cite enabled.

The version in MELPA is still what I call "Version 2" with the old syntax everyone is used to. There is a tagged version at 2.0.0 in MELPA Stable, and the Version 2 branch can be found at https://github.com/jkitchin/org-ref/tree/org-ref-2, for anyone who wants to keep the old behavior.

It is conceivable I might finally be able to split out two new packages for org-ref-ivy and org-ref-helm that do properly install their dependencies, but I am going to wait until it is ready to merge to worry about that.

Until then, if you are the bleeding edge type, give Version 3 a spin, and let me know of any issues.

[bdarcus]

So what are your plans for org-ref-cite? As you know, I've been working on including it in Doom. Should I remove it?

[bdarcus]

Beyond that question, a few comments for the record.

org-cite and org itself do not provide adequate cross-referencing (or index/glossary functions) for scientific publishing, so it is clear that the rest of org-ref also still has a place in this ecosystem.

Obviously cross-referencing is is an issue, though as I and others have said, this is not a limitation of org-cite.

More importantly, I hope you and/or others will propose upstream improvements to org itself to address this. Based on the conversation on the list, it seems there's a straightforward path to do just this.

CSL, on the other hand, uses a generalized citation style like cite/style/variant which is the same across export backends.

You mean "org-cite" here?

The org-cite citation element model is basically biblatex in general (a richer citation model than either natbib or org-ref, whose features are necessary in many areas of the social sciences, humanities, law), plus styles/variants.

The latter does not come from CSL. It's just a way to design the system so citations are less directly tied to the export backends, and to cut down on the number of styles.

The specific styles and variants we came up with were based on evaluating natbib and BibLaTeX primarily, and generalizing them so they would also fit well beyond that (most notably in CSL).

But that was a first stab at it, it's not easy (mostly because of inconsistencies in biblatex and natbib), and the intention was to iterate it based on real-world feedback. So people shouldn't think of this as set it stone.

I found it was not easy to map these onto the full set of natbib/biblatex citation commands, and not easy to remember that mapping while writing.

This is mostly a UI issue though.

We've previously experimented with using annotations to display the actual command.

I don't see why that's insufficient, but if you want the candidates themselves to be the commands (for example, so you can complete against them), you can simply do that, and look up the corresponding org-cite style.

(defvar cmd-style
  '(("citet"  . "/t")
    ("citep" . "/")
    ("Citet" . "/t/c")))

(defun foo--style-choose (prompt)
  (let ((choice
          (completing-read prompt cmd-style)))
    (cdr (assoc choice cmd-style))))

(foo--style-choose "Natbib command: ")

Of course, "the full set" goes back to my point above.

If you really feel that's necessary, I would probably itemize commands that are not supported, and then:

1. if it makes sense, suggest adding style and/or variant mappings for the included processors (natbib, biblatex, csl)
2. just include whatever else; the worst thing that happens is they only work as expected in one specific processor, which isn't any different than org-ref

[jkitchin]

So what are your plans for org-ref-cite? As you know, I've been working on including it in Doom. Should I remove it?

I don't know what my plans for it are. I am not going to remove the repo or anything, so there is no reason to remove it. I don't have further development plans for it at this time. It seems complete in its goal.

[bdarcus]

Fair enough.

I removed it; can always add it back later if and when its future becomes more clear (for example, if you add it to MELPA, commit to support it, etc.).

[jkitchin]

I think I am about ready to merge the version 3 branch back to master. When that happens, there will be a new release on MELPA. In it's current form, there will be some breakage, because I eliminated nearly all the org-ref-* variables that were basically duplicating bibtex-completion variables. I may push an intermediate version of the master branch that prints a warning if these variables are being used.

I decided to use a slightly different syntax than org-cite, so the cite links will look like cite:&key in org-ref. That way it will never be confusing what is being used, and there will never be a conflict with fontification (org-cite too aggressively fontifies org-ref links at the moment when they use @keys). Otherwise, org-ref will provide the same functionality supporting pre/postnotes, and using CSL for non-latex exports.

[jkitchin]

I think I am going to merge org-ref-3 into the master branch on Monday, Oct 18.

I pushed a commit to provide people with some warning about the transition.

[bdarcus]

Notwithstanding the possibly significant legacy document challenges, is there any reason glossaries, cross-references etc. couldn't be handled with small, discrete packages independently of citations?

And while I think of it, functions to convert to and from org-cite?

[jkitchin]

Sure they could be. I think they are all part of one package though, and that is how I develop and support them.

On Sat, Oct 9, 2021 at 12:55 PM bdarcus @.***> wrote:

Notwithstanding the possibly significant legacy document challenges, is there any reason glossaries, cross-references etc. couldn't be handled with small, discrete packages independently of citations?

And while I think of it, functions to convert to and from org-cite?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jkitchin/org-ref/issues/892#issuecomment-939327252, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMJCVSGJQABM26KM7YWQHLUGBXZVANCNFSM5CZ5VMJA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- John

---

Professor John Kitchin (he/him/his) Doherty Hall A207F Department of Chemical Engineering Carnegie Mellon University Pittsburgh, PA 15213 412-268-7803 @johnkitchin http://kitchingroup.cheme.cmu.edu

[bdarcus]

I understand; was just wondering, certainly in terms of thinking about possible org enhancements.

[dschrempf]

Thank you for writing this up. I noticed that much of the functionality was missing in org-cite and was confused about the status of org-ref. I think some of the information provided here should be made accessible in the README. In particular, the fact that org-ref is further maintained and that it can be used in parallel with org-cite.

Thank you for working on this project!

[dschrempf]

One more thing: Maybe bibtex-actions (@bdarcus) could be mentioned as an alternative completion method to helm and ivy.

[bdarcus]

One more thing: Maybe bibtex-actions (@bdarcus) could be mentioned as an alternative completion method to helm and ivy.

Bibtex-actions fully supports org-cite. But I'm not really sure how it might be used in org-ref 3, nor am I interested in working on this myself.

Org-cite has the really modular and flexible "processor" component system, which is what I use. I imagine for OR3, someone would need to write a little function to do something similar to insert citations. Exactly what that would look like would depend on the OR3 design. It could be, for example, that it supports a similar "insert processor" variable, that takes a function with the same signature as an org-cite insert processor.

[jkitchin]

I agree with @bdarcus. I don't think there is an obvious way to use bibtex-actions short of writing a new backend for it. It wouldn't be hard to do, the org-ref-helm and org-ref-ivy files are pretty short. I don't use bibtex-actions myself though, and am unlikely to do that either.

Since I have merged version 3 in, I am going to close this issue.

[dschrempf]

Hm maybe I am getting something wrong here. I use bibtex-actions for all completion functions, and still, I use org-ref for all my citations. I prefer the library of @bdarcus to helm or ivy.

[bdarcus]

Hm maybe I am getting something wrong here. I use bibtex-actions for all completion functions, and still, I use org-ref for all my citations. I prefer the library of @bdarcus to helm or ivy.

To be clear, and maybe this is what you were getting at, you can use bibtex-actions-insert-citation now to insert traditional org-ref citation links.

Sorry for the misunderstanding if that's what you mean. In that case, I agree it couldn't hurt to include a link in the README.

[dschrempf]

Yes, that's what I am using. Sorry for not being clear enough!

[jkitchin]

If you can provide the "recipe" to setup bibtex actions for this, I can add it in to the readme.

Note, you do not have to use helm or ivy with org-ref. The core library provides a completion agnostic insert mechanism that you could wire into selectrum, or some other completion tool. You can also use any other way to insert cite links, so if someone likes ebib, they could use that instead.

[dschrempf]

Hm, there is not much setup involved. I guess I am actually doing what you suggest: Using the completion agnositc insert mechanism. I also set

  (setq bibtex-actions-bibliography `(,my-bibliography-file))
  (setq bibtex-actions-library-paths `(,my-bibliography-papers-dir))
  (setq bibtex-actions-notes-paths `(,my-bibliography-notes-dir))
  (setq bibtex-completion-bibliography `(,my-bibliography-file))
  ;; `org-ref' expects a slash at the end of the path.
  (setq bibtex-completion-library-path (concat my-bibliography-papers-dir "/"))
  (setq bibtex-completion-notes-path my-bibliography-notes-dir))

So that they point to the same files/directories. The my-XXX variables are just strings matching paths to files or directories.

[bdarcus]

It's probably easiest to just link to the repo.

There is the configuration section, but the details may change in the future (now it uses bibtex-completion for the insert citation command).

[japhir]

Are plain ac:tla acronym calls now no longer supported? I get empty acronyms and glossaries on a document that doesn't use the [[ac:tla][TLA]] syntax. Or perhaps the problem stems from somewhere else, it appears that the tables with #+name: acronyms and #+name: glossary don't result in \newglossaryentry headings in the latex export.

I'm on org-ref-20211102.1926

[jkitchin]

@japhir plain ac links aren't currently defined. I don't see them described in https://en.wikibooks.org/wiki/LaTeX/Glossary, and I think I recall they are just aliases for gls I think?

You have to add this hook function I think:

;; Then add org-ref-glossary-before-parsing' and ;;org-ref-acronyms-before-parsing' to the `org-export-before-parsing-hook'.

This should probably be a new issue if those don't work.

[japhir]

Great! Adding the new hooks fixed that part of the problem!

Up next: when I write a prescript but not a postscript, it gets pushed to the end. I can force it to the beginning by appending a ; at the end. Is this the desired behaviour? And if so, can that be clarified a bit in the documentation?

[[parencite:i.e.;&Zachos2001;&Zachos2008;&Westerhold2020]] → (Zachos et al., 2001; Zachos et al., 2008; Westerhold et al., 2020, e.g.) [[parencite:i.e.;&Zachos2001;&Zachos2008;&Westerhold2020;]] → (e.g. Zachos et al., 2001; Zachos et al., 2008; Westerhold et al., 2020)

[jkitchin]

Can you file this as a new issue, with some more details. e.g. how are you exporting (to latex or with CSL)? What bibliography style are you using, etc.