org-ref 3.0?

[jkitchin]

The new citation capability of org-mode is coming soon. That development will make at least the cite capability of org-ref obsolete. I am starting to think about what that means for org-ref, and what the future of it is. I am definitely going to tag a version 2.0 for Melpa stable, which will be like the current version, but with limited support.

The next generation will be a version 3. I will also take this opportunity to remove all the backends but one, and if I can figure out how to make it agnostic and focused on completing-read(-multiple) to do what I want then I will, but if not, it will probably use ivy. I don't have a time frame for when this might be done; it won't start until the org-cite capability is in the main version of org, and I guess it could take up to a year to work out what should be removed, and what should stay.

It is possible I will make a new package like org-ref3 that is eventually going to replace org-ref.

Leaving this issue here for public comment.

[Zilong-Li]

It would be great if org-ref use ivy rather than helm, at least for me. I already replaced all helm thing with ivy in my configuration. But I have to install helm back to use org-ref, which is not favored for me.

[jkitchin]

@tmalsburg I wonder what you are planning for transitioning helm/ivy-bibtex to the new org-cite syntax? Are you planning to:

1. Handle insertion in the new syntax?
2. provide follow processor that uses h/i-bibtex functions?
3. Anything else?

[tmalsburg]

1. Handle insertion in the new syntax?

Yes, switching the default to the new cite syntax makes a lot of sense. Users can still configure helm/ivy-bitex to use other formats if that's what they prefer.

Is there a formal description of the new cite syntax somewhere?

2. provide follow processor that uses h/i-bibtex functions?

I'm not super up-to-date with recent developments in this area. What's a good place to start reading up on follow processors?

3. Anything else?

Not sure. Current helm/ivy-bibtex was largely designed during a time when people weren't using org for serious academic writing. So there may be more room for improvements but I'm not using org for these purposes myself (coauthors prefer LaTeX), so I'll have to rely on the community's input.

[jkitchin]

1. Handle insertion in the new syntax?

Yes, switching the default to the new cite syntax makes a lot of sense. Users can still configure helm/ivy-bitex to use other formats if that's what they prefer.

Is there a formal description of the new cite syntax somewhere?

Not exactly. The best reference now is at https://www.mail-archive.com/emacs-orgmode@gnu.org/msg137770.html. It has taken some playing around to figure it out fully.

1. provide follow processor that uses h/i-bibtex functions?

I'm not super up-to-date with recent developments in this area. What's a good place to start reading up on follow processors?

at the moment it is the docstrings in the new .el files. Out of the box, the follow processor just follows the citation to the bibtex entry. It is the org-cite-basic-goto function in oc-basic.el. I will probably make one that calls a hydra.

1. Anything else?

Not sure. Current helm/ivy-bibtex was largely designed during a time when people weren't using org for serious academic writing. So there may be more room for improvements but I'm not using org for these purposes myself (coauthors prefer LaTeX), so I'll have to rely on the community's input.

[jkitchin]

I have been learning more about org-cite. I think my plan for org-ref will look like this:

1. ivy-bibtex will be used to select candidates from your bibliography, and a custom org-ref function will insert the citations with the same kind of functionality you are used to: a. if there is no citation, insert one b. If there is a citation at the point, append it c. if you want an alternative cite type, you will type C-u and then select the type you want (these may include pre/post text variants)

2. I will add a new org-cite-processor called oc-bibtex.el. This will define a follow function that is a hydra that will use bibtex-completion functions where possible, and org-ref functions if needed, and an activate function that will put a keymap on the cite links, and export functions to support bibtex. It will also define cite styles that look more like the links from org-ref. It will also make the cites green like I am used to. This processor will only be a little different than oc-natbib.el and may even reuse some functions from it.

3. I will probably take this opportunity to separate out the ref links into an org-ref-crossref.el library.

4. I have to think still about doi-utils and arxiv and the libs like glossaries and acronym, etc.

Thinking about this some more, I think we have to have a separate package than org-ref in the future. Otherwise, there won't be a way to support the old link syntax and the new org-cite syntax.

[tmalsburg]

1. ivy-bibtex will be used to select candidates from your bibliography, and a custom org-ref function will insert the citations with the same kind of functionality you are used to:

Sounds good. I'd be okay with including this as the default citation function for org in helm/ivy-bibtex. I'd have implemented something similar anyway I guess.

2. I will add a new org-cite-processor called oc-bibtex.el.

In helm/ivy-bibtex I may offer a follow-processor that simply opens helm/ivy-bibtex search with the entry in question pre-selected, which would then provide quick access to actions.

[bdarcus]

FWIW, you can see how I did insert and follow processors here:

https://github.com/bdarcus/bibtex-actions/blob/main/bibtex-actions-org-cite.el#L49

It was really very easy.

Brief screencast:

https://www.youtube.com/watch?v=gqehXIZcqU8

Aside: you can see the "follow" processor behavior towards the end where I do M-x org-open-at-point, and then contextual actions via embark-act (though this is independent of that oc processor, unless one sets it to use embark-act).

What the insert processor does:

1. substitute bibtex-actions for browsing and selection
2. reuse the bundled oc-basic function for inserting and editing the citations

I'm pretty sure it would be trivial to do for ivy-bibtex and helm-bibtex.

My suggestion is you, @tmalsburg, experiment with a similar approach before writing your own version of 2.

Here's how it works, when doing M-x org-cite-insert:

• if not on a citation, insert full citation from key(s)
• if on a citation key, prompt for replacement key(s)
• if elsewhere on the citation, append key(s)

Really, the basic insert processor is very well done.

I had earlier written an org-cite formatting function, which could be added to bibtex-completion (feel free to borrow it @tmalsburg), but this was before the above, which I think is better.

https://github.com/bdarcus/bibtex-actions/blob/main/bibtex-actions.el#L215

One place that needs improvement is the style selection UI. I have an issue to figure that out here, but happy to brainstorm across projects on this.

https://github.com/bdarcus/bibtex-actions/issues/168

In helm/ivy-bibtex I may offer a follow-processor that simply opens helm/ivy-bibtex search with the entry in question pre-selected, which would then provide quick access to actions.

Hmm ... you may not need helm or ivy here, though the best approach would depend on the UX.

@jkitchin's idea of the hydra-based follow processor could work be an alternative.

So use helm/ivy-bibtex for browsing, but use the org-ref hydra follow processor for access to the functions?

Really depends what the best UI is; going back to helm/ivy UI, or using more directly with hydra.

But the great thing about org-cite is because it's so modular, could provide both options to users easily.

In the end, a user should have the flexibility to mix-and-match:

(setq org-cite-insert-processor 'oc-helm-bibtex) ; helm UI
(setq org-cite-follow-processor 'oc-helm-bibtex) ; open helm-bibtex with pre-selected candidates

(setq org-cite-insert-processor 'oc-helm-bibtex) ; helm UI
(setq org-cite-follow-processor 'oc-org-ref) ; hydra contextual actions

(setq org-cite-insert-processor 'oc-ivy-bibtex) ; ivy UI
(setq org-cite-follow-processor 'oc-bibtex-completion) ; a basic one that uses bibtex-completion-at-point

Aside: if this made sense, I probably wouldn't have a separate oc-bibtex-completion, but instead do something like:

(org-cite-register-processor 'oc-ivy-bibtex
  :insert (org-cite-make-insert-processor
           #'ivy-bibtex-oc-insert
           #'org-cite-basic--complete-style)
  :follow #'bibtex-completion-oc-follow)

As in, there'd be one "follow" function in bibtex-completion, but you could use the same in both oc-ivy-bibtex and oc-helm-bibtex. So:

(setq org-cite-insert-processor 'oc-ivy-bibtex) ; ivy UI
(setq org-cite-follow-processor 'oc-ivy-bibtex) ; this wraps bibtex-completion-at-point

And then for bibtex-actions, users also have choices.

(setq org-cite-insert-processor 'bibtex-actions-org-cite) ; completing-read UI
(setq org-cite-follow-processor 'oc-org-ref) ; ... but hydra at point

(setq org-cite-insert-processor 'bibtex-actions-org-cite)
(setq org-cite-activate-processor 'oc-org-ref) ; I don't have this capability in my processor, so use this instead of `oc-basic`
(setq org-cite-follow-processor 'bibtex-actions-org-cite) ; embark contextual actions

Ideally it would be easy to also mix-and-match different functions across the follow processors, though I haven't thought through how. This is the question I presented here:

https://github.com/jkitchin/org-ref/issues/883

In Embark, command mappings are provided via keymaps, so it's easy to use different keymaps, and so configure which actions are available at point. Not sure about Hydra.

But certainly functions that take a list of keys as input would be a precondition for that. Would be awesome to be able to add an org-ref function, for example, to my embark keymap, and have it "just work."

Suffice to say if this could work, it would be ideal all around: much more flexible for users, with more innovation, and much easier for developers.

[tmalsburg]

Thanks for these pointers, Bruce. Sounds really promising.

[bdarcus]

Your plan sounds good @jkitchin.

Beyond what I already wrote, which was mostly directed to @tmalsburg and helm/ivy-bibtex/bibtex-completion ...

1. ivy-bibtex will be used to select candidates from your bibliography,

If @tmalsburg offers an oc-ivy-bibtex processor that includes insert capabilities, I don't think you need to worry about this, and can just focus on follow (and maybe activate) functionality. The user could then just choose on their own which frontend to use.

I may be missing something about ivy/helm/hydra, but it would be nice for everyone if I'm right! I think I am though, as you're getting the keys from the buffer at point.

2. I will add a new org-cite-processor called oc-bibtex.el. This will define a follow function that is a hydra that will use bibtex-completion functions where possible, and org-ref functions if needed,

I have two questions:

1. is it ok to use that "oc-" prefix outside org proper? I had wondered about that. If yes, I'll change my processor name.
2. this makes sense, though why not call it "oc-org-ref"? Bibtex is super general, and everyone is already familiar with "org-ref"?

and an activate function that will put a keymap on the cite links, and export functions to support bibtex.

I'm not super clear on the bounds of "activate" functionality, but wouldn't these be for the "follow" processor; the hydra?

BTW, on activate processors, see this, which I'm hoping will get added to oc-csl:

https://github.com/andras-simonyi/org-cite-csl-activate

It will also make the cites green like I am used to.

Can't we already change that with the oc faces?

[jkitchin]

See https://www.youtube.com/watch?v=4ta4J20kpmM for the current status of my work. I have a working processor that is pretty close to the citation capability of org-ref now. This isn't super close to being MELPA ready, but is definitely moving in the right direction.

@tmalsburg this uses functions from ivy-bibtex, but doesn't require any modifications to ivy-bibtex at this point.

[bdarcus]

Nice work @jkitchin!

You won't be surprised by what I say in the following, but I think it's worth flagging as a core design question.

You say this in the document that accompanies the source:

This library is primarily written for writing scientific papers that use bibtex (not biblatex) and natbib. The ultimate goal is eventually a LaTeX file that will be submitted to publishers. I think it is better to stay close to the way these files will look, and not to have mentally map a universal style to the natbib commands. I could be wrong about this.

None of us can really know for sure, but this is an assumption with big implications for the modularity of the code.

Right now, you have a monolithic processor, where every piece depends on the other.

• The insert processor depends on the export processor, and the source documents produced with it won't work with any other export processor.
• You can't swap in different insert processors, or use your insert processor elsewhere.
• etc.

I'd encourage you and the community to think about whether this core assumption is really necessary.

There are other ways to address the legitimate "mental mapping" point you raise; for example, annotations (or ivy display transformers?) that show the user what a style input will produce.

So, for example, imagine a completion interface that shows the output mapping directly; where the user sees something like this to choose the style, with the "yellow" indicating the currently active or default target (here natbib is that, but csl is also available, and providing the output preview for that export processor).

... and perhaps can use an approach like org-cite-csl-activate to directly display the latex output on the citation in buffer.

That could avoid the mental mapping challenge (and actually provides more info), while allowing the pieces to be more modular per my points above.

I don't see any reason this can't work.

https://github.com/bdarcus/bibtex-actions/issues/168 https://github.com/bdarcus/bibtex-actions/issues/168#issuecomment-882046878

[bdarcus]

Here's the current state of my WIP code, which shows what I mean, with annotations for the natbib mappings:

So user can easily see what they need (via the annotations), and then filter and select.

Probably as they use it more, they internalize the mappings.

And best of all, the citations will then be portable across export processors. So next step is a column with a CSL preview, for example.

If a user wants to use this and the bibtex-actions UI, they can do:

(setq org-cite-insert-processor 'bibtex-actions-org-cite) ; am still planning to change the name

... or if they want even finer-grained control, they could do this:

(org-cite-register-processor 'janes-custom-processor
  :insert (org-cite-make-insert-processor
           #'oc-helm-bibtex-insert
           #'bibtex-actions-org-cite-select-style)
  :follow #'oc-bibtex-follow)

... which would use (assuming @tmalsburg makes one) helm-bibtex for browsing, my style selector, and the org-ref-derived follow capabilities.

I almost wonder if we need a separate project, call it something like oc-bricks, just for these little mix-and-match components.

PS - my code is using completing-read without anything else; the annotations will work starting with Emacs 27, and grouping with Emacs 28. With earlier versions, you will just get the list of org-cite style and variants.

[bdarcus]

I almost wonder if we need a separate project, call it something like oc-bricks, just for these little mix-and-match components.

Here's what this might look like, with this for the start of an insert processor (based on what I have done so far at bibtex-actions).

https://github.com/bdarcus/oc-blocks

... and this key defcustom:

(defcustom oc-blocks-insert-completion-function 'bibtex-actions-read
  "A completion function that returns a list of citation keys."
  ;; Need to do this ideally without requiring any of these packages, or else not worth it.
  :type '(choice
          (function-item :tag "ivy-bibtex (ivy)" :value ivy-bibtex-read) ; not sure if this is right, but shows the idea
          (function-item :tag "helm-bibtex- (helm)" :value helm-bibtex-read)
          (function-item :tag "bibtex-actions (completing-read)" :value bibtex-actions-read)
          (function :tag "Custom function"))
  :group 'oc-blocks-insert)

Or alternately, the function that uses that is tiny, so could just have parallel insert-key functions and/or files or processors: oc-blocks-select-keys-ivy, oc-blocks-ivy.el, etc.

On the style selection UI, from what I can tell, ivy supports completing-read's annotation-function, so I think completing-read should be enough there, and more compatible.

[bdarcus]

Ivy and compatibility

@jkitchin I finally had a chance to download your code and see what it would take to get at least your follow processor working with my package, which is to say without ivy.

It turns out very little!

All it took was commenting out the require line for ivy-bibtex (I guess would need to require bibtex-completion though), and line 728.

EDIT: Well, "change style" doesn't work since you use ivy there. Are you sure we can't just use completing-read for this, and we can't collaborate on that piece? I don't see what ivy adds here.

Or alternately, remove that command from the hydra, since it's not really necessary there anyway?

Minor issue: your package overwrites the default oc face even if I haven't set it to use your activate processor. So I guess at some point move that setq within the activate processor, or leave it out entirely and include in the documentation?

On the activate processor:

I expect it will work without issue, but I got this non-ivy-related error when trying to move a reference within a citation:

org-shiftselect-error: To use shift-selection with Org mode, customize ‘org-support-shift-select’

Is that something you can address?

Is there something I can do to fix on it my end to test further? "Customize" how?

Good to see the progress, BTW; looking really promising!

[bdarcus]

Extensible activate processors?

One other, broader, question that occurred to me.

One design decision that Nicolas made was there can only be one activate processor. But he did say one can design such processors to be extensible.

Is there a way one could use the oc-bibtex activate processor, but get the fontification and tooltips from this (whose functionality will hopefully be merged with oc-csl.el?

https://github.com/andras-simonyi/org-cite-csl-activate

I asked @andras-simonyi the same question for the reverse; could one plugin oc-bibtex activate at-point functionality somehow to org-cite-csl-activate (though I am guessing there may be some UI challenges with that)?

[bdarcus]

Affixes

An issue with inserting affixes, that I've struggled with as well:

Org-cite borrows the biblatex model here, where there are global affixes, and those specific to individual references. The global are most obviously appropriate for "see" references in author-date and note styles; like:

... (for more on x, see Doe, 2019; Jones, 2020)

If you don't put the initial prefix on the global prefix, and you resort the keys, you have a problem.

OTOH, suffixes are almost always specific to individual references.

Aside: I'm pretty sure oc-natbib is smart about how it handles latex export with these (treating prefix for first reference as global, for example), though have not tested myself. Also, I don't think CSL currently has this distinction, so citeproc-el probably has issues with this case as well, unless Andras figured out some similar workaround.

Not sure what the best solution is, but it's worth thinking about. One possibility is to say prefixes are always global, and suffixes are always local.

Or better: simply consider point; before the first reference, or when on the citation prefix, treat as global. Right now I get this with org-open-at-point if on "cite", which looks like unintended behavior?

[bdarcus]

On names, and breadth vs focus

@jkitchin - I know you and many of your users focus on bibtex/latex.

But I will just observe that the functionality you have so far implemented should be a big improvement for all org-cite users, compared to the bundled processors.

So without much in the way of tweaks, it will be killer for biblatex and CSL users as well.

Just something to think about; whether the oc-bibtex name of the follow and activate functionality really does it justice.

I do know that @joostkremers has a development branch (discussion thread), really far along, that adds support for CSL JSON.

And the cool thing about those discussions: he was planning to provide helper functions so developers don't really need to think about it if they don't need to.

Upshot: in the not-distant future, and once that's merged, parsebib should be able to simultaneously simplify bibtex-completion code, and give it support for CSL JSON.

Which of course means all the packages that depends on it also get that support, including mine.

Maybe you could still have a more focused export processor, if you feel the need, but I don't see any reason to restrict the audience otherwise.

And yeah, I've been thinking about changing the bibtex-actions name, in part for this reason.

PS - I think that's it for the day; sorry for the deluge.

[jkitchin]

Ivy and compatibility

@jkitchin I finally had a chance to download your code and see what it would take to get at least your follow processor working with my package, which is to say without ivy.

It turns out very little!

All it took was commenting out the require line for ivy-bibtex (I guess would need to require bibtex-completion though), and line 728.

EDIT: Well, "change style" doesn't work since you use ivy there. Are you sure we can't just use completing-read for this, and we can't collaborate on that piece? I don't see what ivy adds here.

I did figure out how to do this with completing-read here, so that is not an issue.

Or alternately, remove that command from the hydra, since it's not really necessary there anyway? It is related to discoverability, and it is unlikely to be something everyone remembers so I think it fits in the hydra.

Minor issue: your package overwrites the default oc face even if I haven't set it to use your activate processor. So I guess at some point move that setq within the activate processor, or leave it out entirely and include in the documentation?

I don't think the activate processor is where that should go, that runs every time you fontify a cite. I have moved it out of here for now.

On the activate processor:

I expect it will work without issue, but I got this non-ivy-related error when trying to move a reference within a citation:

org-shiftselect-error: To use shift-selection with Org mode, customize ‘org-support-shift-select’

I don't know why that happens. I have org-support-shift-select set to t, but this works for me even when it is not t. I will have to try in a more vanilla environment later. If you type C-h k and then type S-arrowkey, does it show that it is bound to one of my functions?

Is that something you can address?

Is there something I can do to fix on it my end to test further? "Customize" how?

Good to see the progress, BTW; looking really promising!

[jkitchin]

Affixes

An issue with inserting affixes, that I've struggled with as well:

Org-cite borrows the biblatex model here, where there are global affixes, and those specific to individual references. The global are most obviously appropriate for "see" references in author-date and note styles; like:

... (for more on x, see Doe, 2019; Jones, 2020)

If you don't put the initial prefix on the global prefix, and you resort the keys, you have a problem.

OTOH, suffixes are almost always specific to individual references.

Aside: I'm pretty sure oc-natbib is smart about how it handles latex export with these (treating prefix for first reference as global, for example), though have not tested myself. Also, I don't think CSL currently has this distinction, so citeproc-el probably has issues with this case as well, unless Andras figured out some similar workaround.

Not sure what the best solution is, but it's worth thinking about. One possibility is to say prefixes are always global, and suffixes are always local.

Or better: simply consider point; before the first reference, or when on the citation prefix, treat as global. Right now I get this with org-open-at-point if on "cite", which looks like unintended behavior?

Can you put an example cite like this? I think what you are seeing is that it doesn't know what to follow when you are on the global citation prefix (which makes sense to me) so it is asking for a key to follow.

[bdarcus]

Can you put an example cite like this? I think what you are seeing is that it doesn't know what to follow when you are on the global citation prefix (which makes sense to me) so it is asking for a key to follow.

OIC; yes; that's the case then:

[cite:see @kohn2001;@ali2020 p23;@berlet2000]

Couldn't it assume to follow all three keys, given that the backend bibtex-completion function takes lists of keys?

Maybe some of the options are less appropriate for lists though.

In any case, the other alternative on global affixes, I guess, is if point is on the :, or right after it?

If you type C-h k and then type S-arrowkey, does it show that it is bound to one of my functions?

No. Seems a conflict with evil (I use Doom)?

Key Bindings
evil-org-mode-map <insert-state> C-S-l
evil-org-mode-map <normal-state> C-S-l
org-mode-map S-<right>

[jkitchin]

Can you put an example cite like this? I think what you are seeing is that it doesn't know what to follow when you are on the global citation prefix (which makes sense to me) so it is asking for a key to follow.

OIC; yes; that's the case then:

[cite:see @kohn2001;@ali2020 p23;@berlet2000]

Couldn't it assume to follow all three keys, given that the backend bibtex-completion function takes lists of keys?

Maybe some of the options are less appropriate for lists though.

In any case, the other alternative on global affixes, I guess, is if point is on the :, or right after it?

If you type C-h k and then type S-arrowkey, does it show that it is bound to one of my functions?

No. Seems a conflict with evil (I use Doom)?

Key Bindings
evil-org-mode-map <insert-state> C-S-l
evil-org-mode-map <normal-state> C-S-l
org-mode-map S-<right>

Extensible activate processors?

One other, broader, question that occurred to me.

One design decision that Nicolas made was there can only be one activate processor. But he did say one can design such processors to be extensible.

Is there a way one could use the oc-bibtex activate processor, but get the fontification and tooltips from this (whose functionality will hopefully be merged with oc-csl.el?

https://github.com/andras-simonyi/org-cite-csl-activate

I asked @andras-simonyi the same question for the reverse; could one plugin oc-bibtex activate at-point functionality somehow to org-cite-csl-activate (though I am guessing there may be some UI challenges with that)?

I don't see why you couldn't require that lib, and add (org-cite-csl-activate citation) into an activate function. I already sort of do that in oc-bibtex.el, I call (org-cite-basic-activate citation) and then add some extra properties.

[jkitchin]

This is getting hard to follow.

I am not worried about the name oc-bibtex.el just yet. There are a lot of more general things in it that could be separated out with better names.

There is only one dependency on ivy-read right now, and it is the insert processor. If there is a way to use completing-read and completing read multiple for that, and it is close enough to how ivy-read works I could probably be convinced I don't need it there. I don't know how to do that though.

There is a lot of dependency on bibtex-completion.

[bdarcus]

If there is a way to use completing-read and completing read multiple for that, and it is close enough to how ivy-read works I could probably be convinced I don't need it there. I don't know how to do that though.

Give bibtex-actions a spin with Vertico and consult-completing-read-multiple to see WYT?

https://github.com/bdarcus/bibtex-actions#org-cite

If you like it, you can just use the package of course (via bibtex-actions-read, as I do in my insert processor), or borrow code and ideas if you would prefer avoiding a dependency on that. I added some things that aren't strictly necessary, so it can be relatively simple.

But to be clear, I do think ivy-bibtex users should be able to use their preferred solution here; I just don't think it should be a hard dependency if possible; certainly not in the activate and follow processors.

[jkitchin]

I sort of feel dropping one dependency for another (or multiple ones) is a non-starter for me at this point. If the current capability of the insert processor could be duplicated with no new dependencies, then I would be ok dropping the ivy-read, but that is what I don't see how to do.

the activate and follow processors are completion agnostic in any case (although my follow processor does rely on hydra). There isn't a real way to avoid the hydra dependency IMO, other than to consider a transient menu, like magit uses (which I believe is now part of emacs).

[bdarcus]

I sort of feel dropping one dependency for another (or multiple ones) is a non-starter for me at this point. If the current capability of the insert processor could be duplicated with no new dependencies, then I would be ok dropping the ivy-read, but that is what I don't see how to do.

Does ivy work with completing-read functions? Like, how is the experience using ivy with the oc-basic insert function?

If yes, you could redo the function in completing-read (if you look at early iterations of the bibtex-actions-read function, it was much simpler), remove the dependency, and then users have flexibility to use what they want.

I was mainly recommending vertico and bibtex-actions so you see what's possible.

If not, is there a way to have the choice of the completion function configurable, like I was proposing here?

https://github.com/jkitchin/org-ref/issues/885#issuecomment-884743970

I'm not sure myself; I've struggled figuring out how to make things flexible without introducing dependencies.

the activate and follow processors are completion agnostic in any case (although my follow processor does rely on hydra). There isn't a real way to avoid the hydra dependency IMO, other than to consider a transient menu, like magit uses (which I believe is now part of emacs).

I don't see any problem with the hydra dependency there.

I do something similar with embark, and actually prefer your menu.

I don't have a good handle on transient vs hydra, but I had the sense transient is designed more for interfaces to CLI apps (like git). I could be wrong though.

Edit: this seems good:

https://magit.vc/manual/transient/Comparison-With-Other-Packages.html#Hydra

[bdarcus]

If not, is there a way to have the choice of the completion function configurable, like I was proposing here?

I just tested this with this code, and it works.

I think just needs some error handling in case the function isn't available?

 (if (fboundp 'ivy-bibtex-read)
           (ivy-bibtex-read) ; possible ivy-bibtex works as is?
         (user-error "completion function unavailable"))

[bdarcus]

I'm testing transient menus; here's a gist:

https://gist.github.com/bdarcus/f03a60ec8994f06cbd4b9733b25b7782

Looks pretty good, without any dependencies to install!

Could always have a default menu, but have a defcustom for that too, so that people can swap in a different function if they want?

I may do that in bibtex-actions.

[jkitchin]

The transient menu looks almost like the hydra code. It looks like they basically do the same things, just in slightly different ways.

The only other differences are probably based in my much greater experience with hydra, and how to make it do fancy things like have dynamic menus, different colors in places, etc.

Thanks for sharing this example. I added something that is almost functionally equivalent to the hydra menu I had before.

[bdarcus]

Glad to hear.

Trying to add one to bibtex-actions I think turned up some unintended assumptions that I need to untangle.

One detail on the style selector that's eluding my elisp skills:

How to get the flat list of style+variant strings from org-cite-supported-style.

Do you know how to do that?

[jkitchin]

I guess you want something like this?

#+BEGIN_SRC emacs-lisp :results code
(org-cite-supported-styles '(basic))
#+END_SRC

#+RESULTS:
#+begin_src emacs-lisp
((("author" "a")
  ("caps" "c"))
 (("noauthor" "na")
  ("bare" "b"))
 (("nocite" "n"))
 (("note" "ft")
  ("caps" "c")
  ("bare-caps" "bc"))
 (("numeric" "nb"))
 (("text" "t")
  ("caps" "c")
  ("bare-caps" "bc"))
 (("nil")
  ("caps" "c")
  ("bare-caps" "bc")
  ("bare" "b")))
#+end_src

#+BEGIN_SRC emacs-lisp :results code
(cl-loop for group in (org-cite-supported-styles '(basic)) append
     ;; I guess make all combinations of the first with the rest
     (let ((main (pop group)))
       (when (string= "nil" (first main)) (setf (first main) ""))
       (append (list (first main))
           (cl-loop for rest in group collect
                (format "%s/%s"
                    (first main)
                    (first rest))))))

#+END_SRC

#+RESULTS:
#+begin_src emacs-lisp
("author" "author/caps" "noauthor" "noauthor/bare" "nocite" "note" "note/caps" "note/bare-caps" "numeric" "text" "text/caps" "text/bare-caps" "" "/caps" "/bare-caps" "/bare")
#+end_src

[jkitchin]

I sort of feel dropping one dependency for another (or multiple ones) is a non-starter for me at this point. If the current capability of the insert processor could be duplicated with no new dependencies, then I would be ok dropping the ivy-read, but that is what I don't see how to do.

Does ivy work with completing-read functions? Like, how is the experience using ivy with the oc-basic insert function?

It works fine for the simple selection of an item like completing read. That is not what I need for insertion though. You need to be able to insert a single or multiple citations, and do that with minimal effort because you will do it a lot. So I would expect behavior like pressing RET inserts the currently selected entry and exits, pressing mod-RET (where mod is something like Ctrl, Meta, etc) inserts the selected entry but keeps completion open, and that there is a way to mark entries, so that a final RET inserts them all. This isn't currently possible with the way that ivy works on completing-read I think.

In the oc-basic insert function, ivy sort of works, but not like I described above, and somewhat unintuitively to me (it took me a long time to figure out I have to press Ret on the entries I want to select, and then type C-M-j to exit it and insert the citations. That is too many keypresses to me). That might reflect the difference in what I am used to with ivy, and from not using the regular completion tools, but I guess would reflect the experience of a new person too. None of the usual things with ivy work though, presumably because there is no action associated with the completion, e.g. you cannot do sequential selections with C-M-m, and there are no alternate actions to use. It also does not work to mark some entries, and then press C-M-j.

I see this as making easy things easy for people who just want to insert, and amazing things possible for myself. With ivy, I can, for example while inserting open a pdf or notes of the selected entry (without closing completion) to see if it is what I want, and then go back to completion to insert it or choose something else. I can from a single interface email an entry, find citing references or related references instead of inserting if that is what I need.

It is hard to see why I would want to give that up for the sake of modularity, especially since I will just add it back somewhere else. That is a level of indirection that adds a lot of technical debt in the long run (another lesson learned supporting org-ref).

OTOH, this is only an issue for my insert processor, and you can already choose that to be whatever you want.

If yes, you could redo the function in completing-read (if you look at early iterations of the bibtex-actions-read function, it was much simpler), remove the dependency, and then users have flexibility to use what they want.

I was mainly recommending vertico and bibtex-actions so you see what's possible.

If not, is there a way to have the choice of the completion function configurable, like I was proposing here?

org-ref has had this kind of flexibility for a long time, you could choose between reftex, helm-bibtex, helm-cite, and ivy-cite types of backends for insertion, and it is difficult to maintain and support them. Especially helm is a challenge to support because I don't personally use it anymore, and don't know it very well anymore.

You probably can't (or shouldn't) distribute a package that has options for helm/ivy/etc without making those a dependency for the package or you will get lots of bug reports that it doesn't work. The flip side is you will get lots of complaints of why someone has to download some package they regard as heavy/impure/etc just to use your package. This is not a mistake I would repeat again in the next version of org-ref. I think the majority of users are going to install from MELPA or something equivalent, and will want a reasonable set of defaults, and that it just works.

#885 (comment)

I'm not sure myself; I've struggled figuring out how to make things flexible without introducing dependencies.

the activate and follow processors are completion agnostic in any case (although my follow processor does rely on hydra). There isn't a real way to avoid the hydra dependency IMO, other than to consider a transient menu, like magit uses (which I believe is now part of emacs).

I don't see any problem with the hydra dependency there.

I do something similar with embark, and actually prefer your menu.

I don't have a good handle on transient vs hydra, but I had the sense transient is designed more for interfaces to CLI apps (like git). I could be wrong though.

Edit: this seems good:

https://magit.vc/manual/transient/Comparison-With-Other-Packages.html#Hydra

This is a very fair comparison of them. They are very similar indeed.

[bdarcus]

I guess you want something like this?

Yes; thank you!

In the oc-basic insert function, ivy sort of works, but not like I described above, and somewhat unintuitively to me (it took me a long time to figure out I have to press Ret on the entries I want to select, and then type C-M-j to exit it and insert the citations. That is too many keypresses to me).

I think that's just a limitation of that function, as I found the same thing with the completing-read-based vertico and selectrum. See at the bottom for context.

With ivy, I can, for example while inserting open a pdf or notes of the selected entry (without closing completion) to see if it is what I want, and then go back to completion to insert it or choose something else. I can from a single interface email an entry, find citing references or related references instead of inserting if that is what I need.

For sure; in the new completing world ecosystem, you need Embark for that. I like that in that world, actions are an optional feature, with room for other solutions.

I don't require Embark in bibtex-actions; only bibtex-completion.

But I (currently; it may change) require it in my org-cite package.

You probably can't (or shouldn't) distribute a package that has options for helm/ivy/etc without making those a dependency for the package or you will get lots of bug reports that it doesn't work. The flip side is you will get lots of complaints of why someone has to download some package they regard as heavy/impure/etc just to use your package.

Yes, I really hate that org-ref currently installs helm and ivy when I use neither!

I hear you on the tension though.

It is hard to see why I would want to give that up for the sake of modularity, especially since I will just add it back somewhere else. That is a level of indirection that adds a lot of technical debt in the long run (another lesson learned supporting org-ref).

Yes, that's a reasonable point, but ...

OTOH, this is only an issue for my insert processor, and you can already choose that to be whatever you want.

... would be good then to split the insert processor from the rest, so the ivy dependency is constrained to where it's relevant.

---

Below is on completing-read-multiple, which relates to why Nicolas chose that UI approach; if you're interested ...

Edit: https://github.com/bdarcus/bibtex-actions/pull/163

This was the origins of consult-completing-read-multiple, proposed by its author to workaround limitations of the completing-read-multiple UI for long candidate strings.

It's a better approach to the problem Nicolas was trying to solve; you just use RET to select or deselect one or multiple candidates.

I think it's possible, maybe even likely, that at some point an iteration of that will be included in Emacs.

[bdarcus]

PS - I haven't mentioned it yet, but part of my suggestions here comes from working on org-cite support for the Doom biblio module, and thinking where that should be a year from now.

https://github.com/hlissner/doom-emacs/pull/5212

BTW, Doom is about to replace it's default ivy completion module with vertico, embark et al.

[jkitchin]

Can you put an example cite like this? I think what you are seeing is that it doesn't know what to follow when you are on the global citation prefix (which makes sense to me) so it is asking for a key to follow.

OIC; yes; that's the case then:

[cite:see @kohn2001;@ali2020 p23;@berlet2000]

The see prefix in this example "belongs" to the citation-reference with key @kohn2001, at least belongs in the sense that it is parsed as

(citation-reference
 (:key "kohn2001" :begin 12767 :end 12781 :post-blank 0 :prefix
       (#("see " 0 4
      (:parent #0)))
       :parent
       (citation
    (:style nil :begin 12761 :post-blank 0 :end 12807 :contents-begin 12767 :contents-end 12806 :parent
        (paragraph
         (:begin 12761 :end 12807 :contents-begin 12761 :contents-end 12807 :post-blank 0 :post-affiliated 12761 :parent nil))))))

I don't see a good way to keep "see" at the beginning while you are shifting references around because the way the shift functions work is to reorder the citation-references and then to use org to interpret them:(org-element-interpret-data (oc-bibtex-swap index (- index 1) refs)). If it was really a global prefix, it should be part of the citation element IMO. Otherwise, what should happen if the middle reference also has a prefix? does it become the new global value? what should happen to the old value?

Couldn't it assume to follow all three keys, given that the backend bibtex-completion function takes lists of keys?

It is ambiguous what should happen to me. maybe you might want to pick a subset of the keys for something (which at the moment is a subset of 1). Certainly not all follow actions need a list, e.g. copying, changing style.

Maybe a middle ground is that follow actions should work on the element at point and it is up to the function to decide what to do on a citation vs citation-reference.

Maybe some of the options are less appropriate for lists though.

In any case, the other alternative on global affixes, I guess, is if point is on the :, or right after it?

If you type C-h k and then type S-arrowkey, does it show that it is bound to one of my functions?

No. Seems a conflict with evil (I use Doom)?

Maybe you aren't using my activate function, so the keymap isn't getting put on the cite? If you have your point on a citation-reference, and type M-x describe-text-properties, you can see what keymap is defined. If you click on [Show] you should see what S-left etc is defined as.

Key Bindings
evil-org-mode-map <insert-state> C-S-l
evil-org-mode-map <normal-state> C-S-l
org-mode-map S-<right>

[bdarcus]

I don't see a good way to keep "see" at the beginning while you are shifting references around ...

Right; that's why I was saying to get it inserted as global from the beginning, before shifting.

If it was really a global prefix, it should be part of the citation element IMO.

Yes, it should.

That's what I was meaning to say; that the command to modify the affixes should insert as global if point is on the citation prefix.

That should be conceptually clean for the user; right? And of course, resorting would work fine.

Maybe a middle ground is that follow actions should work on the element at point and it is up to the function to decide what to do on a citation vs citation-reference.

I agree it's not straightforward, and this might indeed be a reasonable approach.

Maybe you aren't using my activate function, so the keymap isn't getting put on the cite?

I'll give it another try in the next few days.

[jkitchin]

For anyone following this, I have refactored oc-bibtex.el into https://github.com/jkitchin/scimax/tree/master/org-ref-cite.

It provides activate, insert, follow and export functions, and the main package defines an org-ref-cite processor that combines these.

activate - adds a keymap to citations insert - uses bibtex-completion for candidates, and ivy-read for selection follow - provide a hydra menu of bibtex-completion and biblio functions export - adapted from oc-natbib.

There is an advice library that makes local bibliographies have precedence over global ones, and that turns flyspell off on citations.

This separates out things into separate files, which might help with modularity. I am going to leave the org-ref3 directory for a while, but it will be deprecated and eventually removed.

[bdarcus]

Nice work, and nice name :-)

I'll see if I can hook it up to the doom PR later today, just to test it all.

EDIT: I did get pretty far, but am still working on the use-package (well, doom's use-package/straight macro) incantations to get everything to load correctly, conditioned on completion framework, etc.

On the modularity, it might (need to test more) be better to combine insert and follow files into a single org-ref-cite-activate-follow (or maybe org-ref-cite-buffer?) processor so one can install and load that independently of the rest.

[bdarcus]

PS - have you considered just putting it on a branch of the org-ref repo, or creating a new repo specific to this?

Or does that come later?

[jkitchin]

PS - have you considered just putting it on a branch of the org-ref repo, or creating a new repo specific to this? Or does that come later?

That will be later. I have not decided what the best path forward is.

I may keep it separate from org-ref. I have to have both of them available separately to support all the org-files I (and others) have that use the org-ref cite link syntax. Plus too many people bitterly complain about all the dependencies of org-ref already.

In any case, I don't plan to make that kind of change until 1) org 9.5 is out and easily installable, and 2) I have used it long enough to think it is ready for others to use from a package manager.

[bdarcus]

There is an advice library that makes local bibliographies have precedence over global ones ...

Per my email to the org list, I think we should get this changed to be default behavior.

I still think one should be able to use the global bib with the local, but one should explicitly turn that on.

[jkitchin]

If it is changed so my advice isn't necessary I am happy to remove it. The default everywhere else in org is that local properties supercede more global ones. I don't think you should need a file local variable to make this the case.

I also hope the flyspell advice will be integrated into org too so I can take that out.

[jkitchin]

I decided to go ahead and make a standalone repo for this: https://github.com/jkitchin/org-ref-cite. If there are any issues with it, we can track them in https://github.com/jkitchin/org-ref-cite/issues.

I am going to close this issue.

[bdarcus]

And you got your first PR!

Just to report back here (doesn't require an issue), the refactor does allow me to configure the installation in doom such that it does what I want. So thank you!

(when (featurep! :completion vertico)
  (package! bibtex-actions :pin "6e3a194c3ab655693f8194be78542366755c58c9")
  (package! org-ref-cite :recipe
    (:host github
     :repo "jkitchin/org-ref-cite"
     ;; exclude the main file with package-requires line, since we don't 
     ;; need it, and don't want ivy, etc installed
     :files ("*.el" (:exclude "org-ref-cite.el")))))
``